2023-04-17 12:56:47,100 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507 2023-04-17 12:56:47,114 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-17 12:56:47,132 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-17 12:56:47,133 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f, deleteOnExit=true 2023-04-17 12:56:47,133 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-17 12:56:47,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/test.cache.data in system properties and HBase conf 2023-04-17 12:56:47,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/hadoop.tmp.dir in system properties and HBase conf 2023-04-17 12:56:47,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/hadoop.log.dir in system properties and HBase conf 2023-04-17 12:56:47,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-17 12:56:47,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-17 12:56:47,135 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-17 12:56:47,246 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-17 12:56:47,569 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-17 12:56:47,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-17 12:56:47,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-17 12:56:47,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-17 12:56:47,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 12:56:47,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-17 12:56:47,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-17 12:56:47,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 12:56:47,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 12:56:47,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-17 12:56:47,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/nfs.dump.dir in system properties and HBase conf 2023-04-17 12:56:47,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/java.io.tmpdir in system properties and HBase conf 2023-04-17 12:56:47,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 12:56:47,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-17 12:56:47,577 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-17 12:56:48,014 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 12:56:48,018 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 12:56:48,280 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-17 12:56:48,454 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-17 12:56:48,470 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 12:56:48,513 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 12:56:48,559 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/java.io.tmpdir/Jetty_localhost_localdomain_42333_hdfs____3wn4hb/webapp 2023-04-17 12:56:48,709 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:42333 2023-04-17 12:56:48,744 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 12:56:48,744 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 12:56:49,204 WARN [Listener at localhost.localdomain/38149] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 12:56:49,306 WARN [Listener at localhost.localdomain/38149] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-17 12:56:49,327 WARN [Listener at localhost.localdomain/38149] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 12:56:49,333 INFO [Listener at localhost.localdomain/38149] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 12:56:49,341 INFO [Listener at localhost.localdomain/38149] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/java.io.tmpdir/Jetty_localhost_46173_datanode____.en33pq/webapp 2023-04-17 12:56:49,426 INFO [Listener at localhost.localdomain/38149] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46173 2023-04-17 12:56:49,757 WARN [Listener at localhost.localdomain/43083] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 12:56:50,167 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8e9101763355d2a9: Processing first storage report for DS-09032929-f765-4ade-a376-e34520985675 from datanode fabd7e5e-3182-46cc-af5f-125210f1f6a9 2023-04-17 12:56:50,169 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8e9101763355d2a9: from storage DS-09032929-f765-4ade-a376-e34520985675 node DatanodeRegistration(127.0.0.1:33379, datanodeUuid=fabd7e5e-3182-46cc-af5f-125210f1f6a9, infoPort=38925, infoSecurePort=0, ipcPort=43083, storageInfo=lv=-57;cid=testClusterID;nsid=515148895;c=1681736208088), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-17 12:56:50,169 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8e9101763355d2a9: Processing first storage report for DS-336d2127-0b0c-49ed-b11e-5a843d257998 from datanode fabd7e5e-3182-46cc-af5f-125210f1f6a9 2023-04-17 12:56:50,169 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8e9101763355d2a9: from storage DS-336d2127-0b0c-49ed-b11e-5a843d257998 node DatanodeRegistration(127.0.0.1:33379, datanodeUuid=fabd7e5e-3182-46cc-af5f-125210f1f6a9, infoPort=38925, infoSecurePort=0, ipcPort=43083, storageInfo=lv=-57;cid=testClusterID;nsid=515148895;c=1681736208088), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-17 12:56:50,183 DEBUG [Listener at localhost.localdomain/43083] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507 2023-04-17 12:56:50,305 INFO [Listener at localhost.localdomain/43083] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f/zookeeper_0, clientPort=62647, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-17 12:56:50,316 INFO [Listener at localhost.localdomain/43083] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62647 2023-04-17 12:56:50,324 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:50,327 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:50,951 INFO [Listener at localhost.localdomain/43083] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16 with version=8 2023-04-17 12:56:50,951 INFO [Listener at localhost.localdomain/43083] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/hbase-staging 2023-04-17 12:56:51,264 INFO [Listener at localhost.localdomain/43083] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-17 12:56:51,653 INFO [Listener at localhost.localdomain/43083] client.ConnectionUtils(127): master/jenkins-hbase17:0 server-side Connection retries=45 2023-04-17 12:56:51,678 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:51,678 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:51,678 INFO [Listener at localhost.localdomain/43083] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 12:56:51,679 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:51,679 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 12:56:51,798 INFO [Listener at localhost.localdomain/43083] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 12:56:51,861 DEBUG [Listener at localhost.localdomain/43083] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-17 12:56:51,944 INFO [Listener at localhost.localdomain/43083] ipc.NettyRpcServer(120): Bind to /136.243.18.41:44317 2023-04-17 12:56:51,956 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:51,958 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:51,975 INFO [Listener at localhost.localdomain/43083] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44317 connecting to ZooKeeper ensemble=127.0.0.1:62647 2023-04-17 12:56:52,409 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:443170x0, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 12:56:52,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44317-0x1019766c69b0000 connected 2023-04-17 12:56:53,201 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 12:56:53,202 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 12:56:53,206 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 12:56:53,216 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44317 2023-04-17 12:56:53,216 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44317 2023-04-17 12:56:53,216 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44317 2023-04-17 12:56:53,220 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44317 2023-04-17 12:56:53,221 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44317 2023-04-17 12:56:53,226 INFO [Listener at localhost.localdomain/43083] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16, hbase.cluster.distributed=false 2023-04-17 12:56:53,280 INFO [Listener at localhost.localdomain/43083] client.ConnectionUtils(127): regionserver/jenkins-hbase17:0 server-side Connection retries=45 2023-04-17 12:56:53,281 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:53,281 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:53,281 INFO [Listener at localhost.localdomain/43083] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 12:56:53,281 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 12:56:53,281 INFO [Listener at localhost.localdomain/43083] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 12:56:53,285 INFO [Listener at localhost.localdomain/43083] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 12:56:53,288 INFO [Listener at localhost.localdomain/43083] ipc.NettyRpcServer(120): Bind to /136.243.18.41:40215 2023-04-17 12:56:53,290 INFO [Listener at localhost.localdomain/43083] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-17 12:56:53,294 DEBUG [Listener at localhost.localdomain/43083] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-17 12:56:53,295 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:53,298 INFO [Listener at localhost.localdomain/43083] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:53,299 INFO [Listener at localhost.localdomain/43083] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40215 connecting to ZooKeeper ensemble=127.0.0.1:62647 2023-04-17 12:56:53,404 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:402150x0, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 12:56:53,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40215-0x1019766c69b0001 connected 2023-04-17 12:56:53,408 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 12:56:53,410 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 12:56:53,412 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 12:56:53,412 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40215 2023-04-17 12:56:53,413 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40215 2023-04-17 12:56:53,413 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40215 2023-04-17 12:56:53,414 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40215 2023-04-17 12:56:53,417 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40215 2023-04-17 12:56:53,419 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:53,447 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 12:56:53,448 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:53,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 12:56:53,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 12:56:53,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:53,559 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 12:56:53,560 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase17.apache.org,44317,1681736211069 from backup master directory 2023-04-17 12:56:53,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 12:56:53,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:53,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 12:56:53,565 WARN [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 12:56:53,565 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:53,568 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-17 12:56:53,570 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-17 12:56:54,121 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/hbase.id with ID: 6d9b17a9-de20-4ee5-b7fb-7fc19b162d76 2023-04-17 12:56:54,194 INFO [master/jenkins-hbase17:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 12:56:54,245 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:54,843 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1a61a90a to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:56:54,977 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61a77c02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:56:55,007 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:56:55,010 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-17 12:56:55,028 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-17 12:56:55,028 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-17 12:56:55,030 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:56:55,035 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:56:55,037 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 12:56:55,158 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store-tmp 2023-04-17 12:56:55,190 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:55,191 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 12:56:55,191 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:56:55,191 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:56:55,191 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 12:56:55,191 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:56:55,191 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:56:55,191 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 12:56:55,193 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/WALs/jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:55,214 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C44317%2C1681736211069, suffix=, logDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/WALs/jenkins-hbase17.apache.org,44317,1681736211069, archiveDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/oldWALs, maxLogs=10 2023-04-17 12:56:55,280 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK] 2023-04-17 12:56:55,288 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:56:55,591 INFO [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/WALs/jenkins-hbase17.apache.org,44317,1681736211069/jenkins-hbase17.apache.org%2C44317%2C1681736211069.1681736215225 2023-04-17 12:56:55,592 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:56:55,593 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:56:55,594 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:55,598 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,599 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,664 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,671 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-17 12:56:55,697 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-17 12:56:55,715 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:55,722 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,724 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,746 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 12:56:55,751 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:56:55,753 INFO [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69407925, jitterRate=0.034258678555488586}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 12:56:55,753 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 12:56:55,755 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-17 12:56:55,778 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-17 12:56:55,778 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-17 12:56:55,781 INFO [master/jenkins-hbase17:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-17 12:56:55,783 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-17 12:56:55,833 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 49 msec 2023-04-17 12:56:55,833 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-17 12:56:55,866 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-17 12:56:55,874 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-17 12:56:55,905 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-17 12:56:55,909 INFO [master/jenkins-hbase17:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-17 12:56:55,958 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-17 12:56:55,964 INFO [master/jenkins-hbase17:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-17 12:56:55,967 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-17 12:56:55,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:55,980 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-17 12:56:55,980 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-17 12:56:55,997 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-17 12:56:56,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 12:56:56,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 12:56:56,050 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:56,052 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase17.apache.org,44317,1681736211069, sessionid=0x1019766c69b0000, setting cluster-up flag (Was=false) 2023-04-17 12:56:56,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:56,255 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-17 12:56:56,257 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:56,268 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:56,275 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-17 12:56:56,278 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:56:56,281 WARN [master/jenkins-hbase17:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.hbase-snapshot/.tmp 2023-04-17 12:56:56,368 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(952): ClusterId : 6d9b17a9-de20-4ee5-b7fb-7fc19b162d76 2023-04-17 12:56:56,372 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-17 12:56:56,427 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-17 12:56:56,435 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-17 12:56:56,435 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=5, maxPoolSize=5 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase17:0, corePoolSize=10, maxPoolSize=10 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-04-17 12:56:56,436 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,437 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681736246437 2023-04-17 12:56:56,438 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-17 12:56:56,442 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 12:56:56,442 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-17 12:56:56,446 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 12:56:56,447 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-17 12:56:56,450 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-17 12:56:56,451 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-17 12:56:56,455 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-17 12:56:56,455 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-17 12:56:56,455 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-17 12:56:56,455 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-17 12:56:56,456 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,457 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-17 12:56:56,459 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-17 12:56:56,459 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-17 12:56:56,461 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-17 12:56:56,461 INFO [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-17 12:56:56,463 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1681736216462,5,FailOnTimeoutGroup] 2023-04-17 12:56:56,463 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1681736216463,5,FailOnTimeoutGroup] 2023-04-17 12:56:56,463 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,463 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-17 12:56:56,465 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,465 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,542 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-17 12:56:56,544 DEBUG [RS:0;jenkins-hbase17:40215] zookeeper.ReadOnlyZKClient(139): Connect 0x65686db2 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:56:56,656 DEBUG [RS:0;jenkins-hbase17:40215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6baf8f6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:56:56,657 DEBUG [RS:0;jenkins-hbase17:40215] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f9006c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-04-17 12:56:56,677 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase17:40215 2023-04-17 12:56:56,680 INFO [RS:0;jenkins-hbase17:40215] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-17 12:56:56,680 INFO [RS:0;jenkins-hbase17:40215] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-17 12:56:56,680 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1023): About to register with Master. 2023-04-17 12:56:56,683 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase17.apache.org,44317,1681736211069 with isa=jenkins-hbase17.apache.org/136.243.18.41:40215, startcode=1681736213280 2023-04-17 12:56:56,696 DEBUG [RS:0;jenkins-hbase17:40215] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-17 12:56:56,740 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:36385, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-17 12:56:56,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.ServerManager(394): Registering regionserver=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,762 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16 2023-04-17 12:56:56,763 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:38149 2023-04-17 12:56:56,763 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-17 12:56:56,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 12:56:56,854 DEBUG [RS:0;jenkins-hbase17:40215] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,854 WARN [RS:0;jenkins-hbase17:40215] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 12:56:56,859 INFO [RS:0;jenkins-hbase17:40215] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 12:56:56,859 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/WALs/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,863 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase17.apache.org,40215,1681736213280] 2023-04-17 12:56:56,875 DEBUG [RS:0;jenkins-hbase17:40215] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,875 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 12:56:56,876 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 12:56:56,877 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16 2023-04-17 12:56:56,892 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-17 12:56:56,902 INFO [RS:0;jenkins-hbase17:40215] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-17 12:56:56,925 INFO [RS:0;jenkins-hbase17:40215] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-17 12:56:56,929 INFO [RS:0;jenkins-hbase17:40215] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-17 12:56:56,929 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,931 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-17 12:56:56,938 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,938 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,939 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,939 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,939 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,939 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,940 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase17:0, corePoolSize=2, maxPoolSize=2 2023-04-17 12:56:56,940 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,940 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,940 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,940 DEBUG [RS:0;jenkins-hbase17:40215] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase17:0, corePoolSize=1, maxPoolSize=1 2023-04-17 12:56:56,941 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,942 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,942 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,969 INFO [RS:0;jenkins-hbase17:40215] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-17 12:56:56,973 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,40215,1681736213280-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:56,993 INFO [RS:0;jenkins-hbase17:40215] regionserver.Replication(203): jenkins-hbase17.apache.org,40215,1681736213280 started 2023-04-17 12:56:56,993 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1638): Serving as jenkins-hbase17.apache.org,40215,1681736213280, RpcServer on jenkins-hbase17.apache.org/136.243.18.41:40215, sessionid=0x1019766c69b0001 2023-04-17 12:56:56,994 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-17 12:56:56,994 DEBUG [RS:0;jenkins-hbase17:40215] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,994 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,40215,1681736213280' 2023-04-17 12:56:56,994 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:56:56,995 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:56:56,995 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-17 12:56:56,995 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-17 12:56:56,996 DEBUG [RS:0;jenkins-hbase17:40215] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:56,996 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase17.apache.org,40215,1681736213280' 2023-04-17 12:56:56,996 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-17 12:56:56,997 DEBUG [RS:0;jenkins-hbase17:40215] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-17 12:56:56,998 DEBUG [RS:0;jenkins-hbase17:40215] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-17 12:56:56,998 INFO [RS:0;jenkins-hbase17:40215] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-17 12:56:56,998 INFO [RS:0;jenkins-hbase17:40215] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-17 12:56:57,114 INFO [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C40215%2C1681736213280, suffix=, logDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/WALs/jenkins-hbase17.apache.org,40215,1681736213280, archiveDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/oldWALs, maxLogs=32 2023-04-17 12:56:57,133 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK] 2023-04-17 12:56:57,138 INFO [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/WALs/jenkins-hbase17.apache.org,40215,1681736213280/jenkins-hbase17.apache.org%2C40215%2C1681736213280.1681736217117 2023-04-17 12:56:57,138 DEBUG [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:56:57,298 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:57,301 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 12:56:57,304 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/info 2023-04-17 12:56:57,305 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 12:56:57,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:57,307 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 12:56:57,311 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/rep_barrier 2023-04-17 12:56:57,312 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 12:56:57,313 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:57,313 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 12:56:57,316 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/table 2023-04-17 12:56:57,317 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 12:56:57,318 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:57,320 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740 2023-04-17 12:56:57,321 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740 2023-04-17 12:56:57,325 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:56:57,327 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 12:56:57,350 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:56:57,352 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70073596, jitterRate=0.04417794942855835}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:56:57,352 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 12:56:57,352 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 12:56:57,352 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 12:56:57,352 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 12:56:57,352 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 12:56:57,352 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 12:56:57,354 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 12:56:57,354 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 12:56:57,360 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 12:56:57,360 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-17 12:56:57,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-17 12:56:57,384 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-17 12:56:57,387 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-17 12:56:57,539 DEBUG [jenkins-hbase17:44317] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-17 12:56:57,542 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,40215,1681736213280, state=OPENING 2023-04-17 12:56:57,556 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-17 12:56:57,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:57,558 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 12:56:57,561 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:56:57,791 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:57,794 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-17 12:56:57,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:47310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-17 12:56:57,812 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-17 12:56:57,812 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 12:56:57,816 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase17.apache.org%2C40215%2C1681736213280.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/WALs/jenkins-hbase17.apache.org,40215,1681736213280, archiveDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/oldWALs, maxLogs=32 2023-04-17 12:56:57,834 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK] 2023-04-17 12:56:57,878 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/WALs/jenkins-hbase17.apache.org,40215,1681736213280/jenkins-hbase17.apache.org%2C40215%2C1681736213280.meta.1681736217818.meta 2023-04-17 12:56:57,879 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:56:57,879 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:56:57,881 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-17 12:56:58,088 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-17 12:56:58,096 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-17 12:56:58,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-17 12:56:58,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:58,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-17 12:56:58,264 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-17 12:56:58,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 12:56:58,269 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/info 2023-04-17 12:56:58,270 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/info 2023-04-17 12:56:58,270 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 12:56:58,271 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:58,271 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 12:56:58,273 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/rep_barrier 2023-04-17 12:56:58,273 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/rep_barrier 2023-04-17 12:56:58,274 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 12:56:58,275 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:58,275 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 12:56:58,300 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/table 2023-04-17 12:56:58,301 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/table 2023-04-17 12:56:58,301 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 12:56:58,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:58,306 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740 2023-04-17 12:56:58,309 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740 2023-04-17 12:56:58,313 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:56:58,319 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 12:56:58,326 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70403163, jitterRate=0.04908888041973114}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:56:58,326 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 12:56:58,336 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681736217784 2023-04-17 12:56:58,355 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-17 12:56:58,356 INFO [RS_OPEN_META-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-17 12:56:58,356 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase17.apache.org,40215,1681736213280, state=OPEN 2023-04-17 12:56:58,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-17 12:56:58,359 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 12:56:58,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-17 12:56:58,365 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase17.apache.org,40215,1681736213280 in 798 msec 2023-04-17 12:56:58,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-17 12:56:58,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 994 msec 2023-04-17 12:56:58,378 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 2.0010 sec 2023-04-17 12:56:58,378 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681736218378, completionTime=-1 2023-04-17 12:56:58,379 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-17 12:56:58,379 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-17 12:56:58,454 DEBUG [hconnection-0x8531fa5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:56:58,459 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:47318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:56:58,478 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-17 12:56:58,478 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681736278478 2023-04-17 12:56:58,478 INFO [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681736338478 2023-04-17 12:56:58,478 INFO [master/jenkins-hbase17:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 99 msec 2023-04-17 12:56:58,514 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44317,1681736211069-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:58,515 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44317,1681736211069-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:58,515 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44317,1681736211069-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:58,516 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase17:44317, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:58,517 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-17 12:56:58,527 DEBUG [master/jenkins-hbase17:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-17 12:56:58,538 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-17 12:56:58,540 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 12:56:58,553 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-17 12:56:58,556 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:56:58,560 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:56:58,588 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:58,591 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a empty. 2023-04-17 12:56:58,592 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:58,592 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-17 12:56:59,031 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-17 12:56:59,034 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7a30f634bdc6e3e5b973c8313abab69a, NAME => 'hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:56:59,052 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:59,053 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 7a30f634bdc6e3e5b973c8313abab69a, disabling compactions & flushes 2023-04-17 12:56:59,053 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,053 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,053 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. after waiting 0 ms 2023-04-17 12:56:59,053 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,053 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,053 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 7a30f634bdc6e3e5b973c8313abab69a: 2023-04-17 12:56:59,059 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:56:59,086 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681736219067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736219067"}]},"ts":"1681736219067"} 2023-04-17 12:56:59,115 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:56:59,117 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:56:59,122 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736219117"}]},"ts":"1681736219117"} 2023-04-17 12:56:59,126 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-17 12:56:59,224 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7a30f634bdc6e3e5b973c8313abab69a, ASSIGN}] 2023-04-17 12:56:59,227 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7a30f634bdc6e3e5b973c8313abab69a, ASSIGN 2023-04-17 12:56:59,228 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7a30f634bdc6e3e5b973c8313abab69a, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:56:59,379 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7a30f634bdc6e3e5b973c8313abab69a, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:59,380 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681736219379"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736219379"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736219379"}]},"ts":"1681736219379"} 2023-04-17 12:56:59,388 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 7a30f634bdc6e3e5b973c8313abab69a, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:56:59,548 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,549 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7a30f634bdc6e3e5b973c8313abab69a, NAME => 'hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:56:59,549 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:56:59,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,550 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,553 INFO [StoreOpener-7a30f634bdc6e3e5b973c8313abab69a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,555 DEBUG [StoreOpener-7a30f634bdc6e3e5b973c8313abab69a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/info 2023-04-17 12:56:59,556 DEBUG [StoreOpener-7a30f634bdc6e3e5b973c8313abab69a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/info 2023-04-17 12:56:59,556 INFO [StoreOpener-7a30f634bdc6e3e5b973c8313abab69a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7a30f634bdc6e3e5b973c8313abab69a columnFamilyName info 2023-04-17 12:56:59,557 INFO [StoreOpener-7a30f634bdc6e3e5b973c8313abab69a-1] regionserver.HStore(310): Store=7a30f634bdc6e3e5b973c8313abab69a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:56:59,560 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,561 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,568 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:56:59,572 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:56:59,573 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 7a30f634bdc6e3e5b973c8313abab69a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65535752, jitterRate=-0.023441195487976074}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 12:56:59,573 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 7a30f634bdc6e3e5b973c8313abab69a: 2023-04-17 12:56:59,576 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a., pid=6, masterSystemTime=1681736219542 2023-04-17 12:56:59,582 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,582 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:56:59,583 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7a30f634bdc6e3e5b973c8313abab69a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:56:59,584 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681736219583"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736219583"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736219583"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736219583"}]},"ts":"1681736219583"} 2023-04-17 12:56:59,596 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-17 12:56:59,596 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 7a30f634bdc6e3e5b973c8313abab69a, server=jenkins-hbase17.apache.org,40215,1681736213280 in 203 msec 2023-04-17 12:56:59,624 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-17 12:56:59,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7a30f634bdc6e3e5b973c8313abab69a, ASSIGN in 372 msec 2023-04-17 12:56:59,628 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:56:59,635 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736219634"}]},"ts":"1681736219634"} 2023-04-17 12:56:59,639 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-17 12:56:59,643 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:56:59,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1020 sec 2023-04-17 12:56:59,656 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-17 12:56:59,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-17 12:56:59,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:56:59,700 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-17 12:56:59,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 12:56:59,770 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 78 msec 2023-04-17 12:56:59,774 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-17 12:56:59,900 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 12:56:59,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 193 msec 2023-04-17 12:57:00,047 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-17 12:57:00,053 INFO [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 6.488sec 2023-04-17 12:57:00,053 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-17 12:57:00,109 INFO [master/jenkins-hbase17:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-17 12:57:00,111 INFO [master/jenkins-hbase17:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-17 12:57:00,111 INFO [master/jenkins-hbase17:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-17 12:57:00,112 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44317,1681736211069-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-17 12:57:00,113 INFO [master/jenkins-hbase17:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase17.apache.org,44317,1681736211069-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-17 12:57:00,115 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x0e23d53f to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:00,126 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-17 12:57:00,253 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70531455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:00,371 DEBUG [hconnection-0x33e568b1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:00,385 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33604, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:00,397 INFO [Listener at localhost.localdomain/43083] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:57:00,421 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=204, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=194, AvailableMemoryMB=3842 2023-04-17 12:57:00,441 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:57:00,447 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:41072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:57:00,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:57:00,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:57:00,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:00,474 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:57:00,477 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:57:00,482 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,484 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 empty. 2023-04-17 12:57:00,486 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-17 12:57:00,487 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:57:00,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 12:57:00,550 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:57:00,554 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f14a4c6d8659d8cf4158b082301a3bb3, NAME => 'TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:57:00,588 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:57:00,588 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f14a4c6d8659d8cf4158b082301a3bb3, disabling compactions & flushes 2023-04-17 12:57:00,589 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:00,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:00,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. after waiting 0 ms 2023-04-17 12:57:00,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:00,589 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:00,589 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:00,594 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:57:00,596 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736220596"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736220596"}]},"ts":"1681736220596"} 2023-04-17 12:57:00,599 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:57:00,601 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:57:00,601 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736220601"}]},"ts":"1681736220601"} 2023-04-17 12:57:00,603 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:57:00,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 12:57:00,607 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, ASSIGN}] 2023-04-17 12:57:00,610 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, ASSIGN 2023-04-17 12:57:00,611 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:57:00,762 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f14a4c6d8659d8cf4158b082301a3bb3, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:00,763 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736220762"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736220762"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736220762"}]},"ts":"1681736220762"} 2023-04-17 12:57:00,768 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 12:57:00,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:00,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f14a4c6d8659d8cf4158b082301a3bb3, NAME => 'TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:57:00,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:57:00,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,986 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,991 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A 2023-04-17 12:57:00,991 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A 2023-04-17 12:57:00,993 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:00,993 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14a4c6d8659d8cf4158b082301a3bb3 columnFamilyName A 2023-04-17 12:57:00,995 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(310): Store=f14a4c6d8659d8cf4158b082301a3bb3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:00,995 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:00,997 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B 2023-04-17 12:57:00,997 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B 2023-04-17 12:57:00,998 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:00,998 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14a4c6d8659d8cf4158b082301a3bb3 columnFamilyName B 2023-04-17 12:57:00,999 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(310): Store=f14a4c6d8659d8cf4158b082301a3bb3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:00,999 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:01,003 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C 2023-04-17 12:57:01,003 DEBUG [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C 2023-04-17 12:57:01,003 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:01,004 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f14a4c6d8659d8cf4158b082301a3bb3 columnFamilyName C 2023-04-17 12:57:01,005 INFO [StoreOpener-f14a4c6d8659d8cf4158b082301a3bb3-1] regionserver.HStore(310): Store=f14a4c6d8659d8cf4158b082301a3bb3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:01,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:01,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:01,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:01,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:57:01,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:01,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:57:01,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened f14a4c6d8659d8cf4158b082301a3bb3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62874578, jitterRate=-0.06309577822685242}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:57:01,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:01,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., pid=11, masterSystemTime=1681736220922 2023-04-17 12:57:01,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:01,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:01,025 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f14a4c6d8659d8cf4158b082301a3bb3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:01,025 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736221025"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736221025"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736221025"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736221025"}]},"ts":"1681736221025"} 2023-04-17 12:57:01,034 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-17 12:57:01,034 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 in 260 msec 2023-04-17 12:57:01,038 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-17 12:57:01,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, ASSIGN in 427 msec 2023-04-17 12:57:01,040 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:57:01,041 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736221041"}]},"ts":"1681736221041"} 2023-04-17 12:57:01,043 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:57:01,046 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:57:01,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 582 msec 2023-04-17 12:57:01,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 12:57:01,111 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-17 12:57:01,117 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x169fe8bf to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,122 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42d2ba41, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,133 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,144 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,148 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:57:01,152 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:41088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:57:01,164 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x202b48c4 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,173 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33e5a282, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,174 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x449cbd3d to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,181 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20e4e4f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,181 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x77443fb0 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,259 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 12:57:01,260 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-17 12:57:01,295 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142c8aea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,296 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x45864842 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,447 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6362e5a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,447 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x4654132d to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,574 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f537ba4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,575 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x29be3c5c to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,592 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ecabb06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,593 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbd52be to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,635 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1439cba9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,637 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3fed686f to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,658 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@655398b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,658 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3e044e22 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:01,665 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a7ef79d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:01,679 DEBUG [hconnection-0x2d45398-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,679 DEBUG [hconnection-0x77cf76aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,681 DEBUG [hconnection-0x7a579215-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,679 DEBUG [hconnection-0x258e48f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,703 DEBUG [hconnection-0x709f9e55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,703 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,706 DEBUG [hconnection-0x67b13574-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,711 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,711 DEBUG [hconnection-0x759f09fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,711 DEBUG [hconnection-0x78372f86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,712 DEBUG [hconnection-0x7e1f6218-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:01,717 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33640, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,719 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,722 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,722 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,722 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33660, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,726 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:01,733 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:01,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:01,750 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:01,750 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:01,763 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:01,763 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:01,764 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:01,764 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:01,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:01,811 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:57:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:01,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:01,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:01,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:01,824 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:01,824 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:01,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736221882, max timestamp=1681736221895Num uniques 859; 2023-04-17 12:57:01,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736221882, max timestamp=1681736221897Num uniques 859; 2023-04-17 12:57:01,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:57:01,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:57:01,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736221882, max timestamp=1681736221897Num uniques 859; 2023-04-17 12:57:01,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:57:01,965 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:01,965 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:01,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:01,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:01,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:01,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:01,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:01,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:01,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:01,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:01,991 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:02,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:02,010 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,016 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:02,016 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:02,016 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:02,017 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:02,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:02,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,019 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:02,019 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:02,053 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,054 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,056 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,056 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5975df11[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:02,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:02,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,068 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,068 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:02,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:02,069 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:02,069 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:02,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,077 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,077 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,077 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:02,082 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:02,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:02,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:02,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:02,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:02,082 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:02,242 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 160 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:02,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736281995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,261 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736282250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736282251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736282252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736282259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,379 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,379 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,379 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,392 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:02,392 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:02,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736282481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/5f01686d8acd4fc2aa4c5cfc0221f918 2023-04-17 12:57:02,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736282482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736282483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,493 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:02,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736282491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736282487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:02,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:02,552 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:02,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,563 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:02,563 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:02,563 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,563 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,564 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,565 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,565 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:02,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:02,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:02,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:02,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,567 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:02,567 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:02,567 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:02,568 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,568 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:02,568 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:02,568 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:02,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:02,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:02,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:02,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,571 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:02,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1000d2[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:02,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:02,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:02,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:02,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:02,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:02,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:02,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:02,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:02,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:02,609 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:02,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:02,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:02,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:02,610 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:02,614 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,614 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,615 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,616 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:02,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,629 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:02,629 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:02,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736282687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736282688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736282695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736282705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:02,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736282709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/05cf3992a4e9499c969d8b92876a111c 2023-04-17 12:57:02,729 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:02,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:02,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:02,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:02,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:02,744 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:02,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:02,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:02,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:02,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:02,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:02,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:02,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:02,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:02,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:02,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:02,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:02,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:02,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:02,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:02,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:02,791 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,792 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,792 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:02,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3dda54[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:02,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:02,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:02,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:02,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:02,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:02,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:02,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:02,804 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:02,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:02,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:02,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:02,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:02,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:02,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:02,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/9f8102c5f8cf409c9ace2b01498aa986 2023-04-17 12:57:02,813 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,813 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,813 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:02,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,820 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,828 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:02,828 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,828 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,831 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:02,831 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:02,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:02,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/5f01686d8acd4fc2aa4c5cfc0221f918 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918 2023-04-17 12:57:02,832 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:02,832 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:02,831 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:02,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:02,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918, entries=150, sequenceid=12, filesize=11.6 K 2023-04-17 12:57:02,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/05cf3992a4e9499c969d8b92876a111c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c 2023-04-17 12:57:02,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c, entries=150, sequenceid=12, filesize=11.6 K 2023-04-17 12:57:02,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/9f8102c5f8cf409c9ace2b01498aa986 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986 2023-04-17 12:57:02,903 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 12:57:02,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986, entries=150, sequenceid=12, filesize=11.6 K 2023-04-17 12:57:02,908 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-17 12:57:02,910 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-17 12:57:02,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f14a4c6d8659d8cf4158b082301a3bb3 in 1107ms, sequenceid=12, compaction requested=false 2023-04-17 12:57:02,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:02,932 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:02,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:02,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:02,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:02,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:02,970 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:02,970 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:02,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:02,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,976 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,976 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:02,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:02,977 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:02,977 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:02,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:02,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:02,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:02,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:02,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:02,981 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:02,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:02,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:02,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:02,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:02,986 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-17 12:57:02,986 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:02,987 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:02,987 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:02,987 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:02,987 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:02,987 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:03,055 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/256138f538bc4e138c316a4558ea8184 2023-04-17 12:57:03,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:03,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:03,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736283091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736283092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736283106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736283113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736283115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736283218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736283218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736283228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736283229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736283237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736283434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736283433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736283435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736283436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736283444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,604 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/c27db47f875b4c4e88d0c57f1ef7c598 2023-04-17 12:57:03,687 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/6b784f02850d44d292ac9e7816ef1229 2023-04-17 12:57:03,710 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/256138f538bc4e138c316a4558ea8184 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184 2023-04-17 12:57:03,725 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184, entries=150, sequenceid=47, filesize=11.6 K 2023-04-17 12:57:03,727 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/c27db47f875b4c4e88d0c57f1ef7c598 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598 2023-04-17 12:57:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736283740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736283742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,762 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598, entries=150, sequenceid=47, filesize=11.6 K 2023-04-17 12:57:03,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736283750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,765 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/6b784f02850d44d292ac9e7816ef1229 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229 2023-04-17 12:57:03,782 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229, entries=150, sequenceid=47, filesize=11.6 K 2023-04-17 12:57:03,784 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for f14a4c6d8659d8cf4158b082301a3bb3 in 801ms, sequenceid=47, compaction requested=false 2023-04-17 12:57:03,784 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:03,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:03,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:03,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:03,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:03,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:03,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:03,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:03,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:03,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736283849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736283854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,947 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:03,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:03,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:03,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:03,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a852e98[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:03,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:03,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736283962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:03,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736283963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:03,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:03,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:03,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:03,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:03,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:04,046 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,046 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:04,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,057 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:57:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:04,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:04,059 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:04,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:04,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:04,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:04,060 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:04,083 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,084 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736284167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736284169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736284250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736284251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,263 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,263 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,263 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,264 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:04,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:04,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:04,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,360 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:04,360 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736284360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/88ed0746516d4bf089658cbe32d84a40 2023-04-17 12:57:04,461 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:04,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:04,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:04,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736284473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736284475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:04,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:04,491 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:04,491 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:04,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:04,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,501 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:04,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:04,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:04,511 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:04,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,514 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:04,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:04,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:04,516 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,517 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:04,517 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:04,517 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:04,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:04,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:04,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:04,525 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,525 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,527 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,527 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:04,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d7feb8e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:04,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:04,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,551 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:04,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:04,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:04,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:04,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,553 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:04,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:04,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:04,566 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:04,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:04,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:04,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:04,566 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:04,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/a2e8aa8fea5a457d8fe8aeab4b3bb261 2023-04-17 12:57:04,589 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,593 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,595 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:04,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:04,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,601 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:04,601 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:04,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,702 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:04,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:04,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:04,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:04,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:04,720 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:04,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,721 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:04,721 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:04,722 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,722 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:04,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:04,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:04,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:04,732 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:04,732 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,733 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:04,740 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:04,740 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:04,743 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,743 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:04,743 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:04,743 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:04,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:04,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:04,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:04,745 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,745 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,746 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,746 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:04,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e47c63[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:04,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:04,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,751 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,751 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:04,751 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:04,751 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:04,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:04,759 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:04,759 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,759 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:04,768 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:04,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:04,776 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:04,776 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:04,776 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:04,776 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:04,776 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:04,779 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,779 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,779 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,780 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:04,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:04,806 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:04,806 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:04,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:04,906 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:04,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:04,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:04,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:04,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:04,914 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:04,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:04,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:04,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,917 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:04,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:04,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:04,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:04,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:04,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:04,941 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:04,941 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:04,941 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,941 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:04,941 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:04,941 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:04,941 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:04,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:04,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:04,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:04,943 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:04,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:04,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@252ee951[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:04,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:04,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:04,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:04,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:04,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:04,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:04,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:04,968 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:04,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:04,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:04,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:04,968 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:04,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,974 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,971 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:04,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:04,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736284982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:04,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736284979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:04,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:04,989 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:04,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:04,990 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:04,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:04,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:04,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:04,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:04,999 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:04,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:04,999 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:04,999 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:04,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:05,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/e8e46002104b4fd083603d10588bf8e6 2023-04-17 12:57:05,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/88ed0746516d4bf089658cbe32d84a40 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40 2023-04-17 12:57:05,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40, entries=200, sequenceid=59, filesize=13.9 K 2023-04-17 12:57:05,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/a2e8aa8fea5a457d8fe8aeab4b3bb261 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261 2023-04-17 12:57:05,103 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:05,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:05,112 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261, entries=150, sequenceid=59, filesize=11.6 K 2023-04-17 12:57:05,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/e8e46002104b4fd083603d10588bf8e6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6 2023-04-17 12:57:05,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:05,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:05,139 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:05,140 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:05,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6, entries=150, sequenceid=59, filesize=11.6 K 2023-04-17 12:57:05,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f14a4c6d8659d8cf4158b082301a3bb3 in 1355ms, sequenceid=59, compaction requested=true 2023-04-17 12:57:05,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:05,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:05,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:05,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:05,152 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,152 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:05,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:05,153 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,153 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:05,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:05,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:05,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:05,154 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:05,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:05,155 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:05,155 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:05,154 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:05,156 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:05,157 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:05,157 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:05,157 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:05,157 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:05,157 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:57:05,157 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:05,158 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:05,158 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:05,158 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:05,158 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:05,158 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:05,164 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:05,166 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:05,166 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:05,167 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.8 K 2023-04-17 12:57:05,170 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:05,171 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:05,171 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 05cf3992a4e9499c969d8b92876a111c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681736221781 2023-04-17 12:57:05,171 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:05,193 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=37.1 K 2023-04-17 12:57:05,194 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c27db47f875b4c4e88d0c57f1ef7c598, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681736221982 2023-04-17 12:57:05,194 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 5f01686d8acd4fc2aa4c5cfc0221f918, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681736221781 2023-04-17 12:57:05,196 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a2e8aa8fea5a457d8fe8aeab4b3bb261, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223105 2023-04-17 12:57:05,196 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 256138f538bc4e138c316a4558ea8184, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681736221982 2023-04-17 12:57:05,197 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 88ed0746516d4bf089658cbe32d84a40, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223086 2023-04-17 12:57:05,253 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#10 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:05,260 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#11 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:05,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:05,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:05,307 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/ec259a3a7b31461196475ec916d89048 2023-04-17 12:57:05,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736285349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736285356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736285368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,382 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/23af23720fe04fa88581ab99494bf338 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/23af23720fe04fa88581ab99494bf338 2023-04-17 12:57:05,383 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/4b1c7cad892f4b859cd4a0df9d43c13c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4b1c7cad892f4b859cd4a0df9d43c13c 2023-04-17 12:57:05,408 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 4b1c7cad892f4b859cd4a0df9d43c13c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:05,408 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:05,408 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736225153; duration=0sec 2023-04-17 12:57:05,409 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:05,409 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:05,413 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:05,413 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:05,413 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:05,414 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.8 K 2023-04-17 12:57:05,415 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9f8102c5f8cf409c9ace2b01498aa986, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681736221781 2023-04-17 12:57:05,416 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6b784f02850d44d292ac9e7816ef1229, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681736221982 2023-04-17 12:57:05,417 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 23af23720fe04fa88581ab99494bf338(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:05,417 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e8e46002104b4fd083603d10588bf8e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223105 2023-04-17 12:57:05,418 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:05,418 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736225153; duration=0sec 2023-04-17 12:57:05,419 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:05,433 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/0493d5a1c9fe4060b16de4a9fa966475 2023-04-17 12:57:05,439 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#13 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:05,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736285459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736285463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,598 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/c48d70fe72f7490983b890600d9606d2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c48d70fe72f7490983b890600d9606d2 2023-04-17 12:57:05,609 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into c48d70fe72f7490983b890600d9606d2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:05,609 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:05,609 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736225153; duration=0sec 2023-04-17 12:57:05,610 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:05,620 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f6ac3a7f4a334a8899550481502034df 2023-04-17 12:57:05,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/ec259a3a7b31461196475ec916d89048 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048 2023-04-17 12:57:05,648 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048, entries=150, sequenceid=79, filesize=11.6 K 2023-04-17 12:57:05,656 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/0493d5a1c9fe4060b16de4a9fa966475 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475 2023-04-17 12:57:05,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736285664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:05,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736285666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,671 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475, entries=150, sequenceid=79, filesize=11.6 K 2023-04-17 12:57:05,676 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f6ac3a7f4a334a8899550481502034df as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df 2023-04-17 12:57:05,730 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df, entries=150, sequenceid=79, filesize=11.6 K 2023-04-17 12:57:05,734 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f14a4c6d8659d8cf4158b082301a3bb3 in 577ms, sequenceid=79, compaction requested=false 2023-04-17 12:57:05,734 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:05,734 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:05,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:05,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:05,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:05,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:05,749 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,749 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:05,749 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:05,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:05,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760cf282[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:05,751 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:05,751 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,770 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:05,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:05,771 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:05,771 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:05,808 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:05,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:05,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:05,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:05,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:05,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:05,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:05,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:05,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:05,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:05,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:05,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:05,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:05,840 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:05,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:05,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:05,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:05,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:05,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:05,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:05,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:05,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:05,842 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:05,842 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:05,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:05,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:05,942 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:05,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:06,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:06,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:06,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:06,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:06,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:06,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:06,069 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:06,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:06,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:06,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/f57054ae48ec4d8897c47294c8cfda16 2023-04-17 12:57:06,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:06,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:06,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:06,108 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:06,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:06,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:06,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:06,110 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,110 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:06,111 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:06,111 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,111 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:06,111 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:06,111 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,111 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:06,111 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736286112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736286112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736286122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736286124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,150 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,150 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,151 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,151 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:06,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d5456[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:06,155 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:06,156 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:06,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:06,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:06,163 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,163 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:06,164 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:06,164 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,164 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,164 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:06,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:06,174 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:06,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:06,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:06,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:06,174 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:06,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736286225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736286226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736286226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736286228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/028a0a8284e8444a8d1de49e8fc80681 2023-04-17 12:57:06,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,257 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,264 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,264 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:06,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:06,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,265 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:06,265 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:06,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,365 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:06,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:06,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:06,369 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:06,369 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:06,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:06,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,421 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736286428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736286431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736286433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:06,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:06,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:06,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:06,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:06,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:06,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:06,548 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,548 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:06,548 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:06,548 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:06,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:06,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:06,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:06,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736286541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,569 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,570 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:06,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@277927da[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:06,571 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:06,571 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,586 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:06,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:06,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:06,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:06,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:06,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:06,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:06,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:06,660 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:06,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:06,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:06,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:06,660 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:06,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,679 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:06,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:06,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,679 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:06,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,679 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:06,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736286733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,779 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:06,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:06,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:06,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:06,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:06,786 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:06,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:06,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:06,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,788 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,788 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:06,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:06,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:06,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:06,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:06,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:06,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:06,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:06,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:06,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:06,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:06,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:06,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:06,799 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,800 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,800 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,800 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:06,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754f3605[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:06,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:06,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:06,804 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,804 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:06,804 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:06,804 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:06,805 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,805 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:06,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:06,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:06,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:06,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:06,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:06,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:06,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:06,811 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,818 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,819 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,820 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:06,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:06,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,825 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,825 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,825 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:06,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:06,831 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:06,831 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:06,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736286845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736286845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:06,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736286853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,931 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:06,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:06,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:06,936 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:06,936 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:06,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:06,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,939 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:06,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:06,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:06,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/c8de98149a5b40fc9ccd3d619ae1256f 2023-04-17 12:57:06,940 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,940 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:06,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:06,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:06,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:06,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:06,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:06,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:06,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:06,946 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:06,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:06,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:06,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/f57054ae48ec4d8897c47294c8cfda16 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16 2023-04-17 12:57:06,960 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,960 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:06,961 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:06,961 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:06,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:06,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:06,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:06,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:06,962 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,963 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,963 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,963 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16, entries=150, sequenceid=96, filesize=11.6 K 2023-04-17 12:57:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/028a0a8284e8444a8d1de49e8fc80681 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681 2023-04-17 12:57:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:06,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@644652cc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:06,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:06,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,982 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:06,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:06,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:06,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:06,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:06,983 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:06,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:06,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:06,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:06,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:06,992 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:06,993 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:06,993 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:06,993 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:06,993 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:06,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681, entries=150, sequenceid=96, filesize=11.6 K 2023-04-17 12:57:06,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/c8de98149a5b40fc9ccd3d619ae1256f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f 2023-04-17 12:57:07,003 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,003 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:07,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,003 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,004 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:07,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:07,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:07,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f, entries=150, sequenceid=96, filesize=11.6 K 2023-04-17 12:57:07,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:07,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:07,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:07,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:07,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,011 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:07,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:07,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:07,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:07,011 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:07,011 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:07,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:07,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for f14a4c6d8659d8cf4158b082301a3bb3 in 955ms, sequenceid=96, compaction requested=true 2023-04-17 12:57:07,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:07,014 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:07,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:07,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:57:07,015 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:07,030 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:07,030 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:07,030 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:07,031 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:07,031 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:07,031 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:07,031 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/23af23720fe04fa88581ab99494bf338, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.9 K 2023-04-17 12:57:07,031 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4b1c7cad892f4b859cd4a0df9d43c13c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.9 K 2023-04-17 12:57:07,032 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 23af23720fe04fa88581ab99494bf338, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223105 2023-04-17 12:57:07,034 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ec259a3a7b31461196475ec916d89048, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681736223817 2023-04-17 12:57:07,034 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4b1c7cad892f4b859cd4a0df9d43c13c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223105 2023-04-17 12:57:07,036 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 0493d5a1c9fe4060b16de4a9fa966475, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681736223817 2023-04-17 12:57:07,036 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f57054ae48ec4d8897c47294c8cfda16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:07,042 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 028a0a8284e8444a8d1de49e8fc80681, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:07,073 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:07,076 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:07,115 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:07,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:07,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:07,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:07,132 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:07,132 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:07,133 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:07,133 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:07,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:07,137 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:07,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:07,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:07,138 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/a642c39bfc1544d7ac24dacf0bdae8c8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/a642c39bfc1544d7ac24dacf0bdae8c8 2023-04-17 12:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:07,141 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:07,149 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:07,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:07,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:07,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:07,151 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:07,151 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:07,152 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:07,155 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into a642c39bfc1544d7ac24dacf0bdae8c8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:07,156 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:07,156 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736227014; duration=0sec 2023-04-17 12:57:07,156 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:07,156 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:07,158 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:07,158 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:07,158 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:07,158 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c48d70fe72f7490983b890600d9606d2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.9 K 2023-04-17 12:57:07,159 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c48d70fe72f7490983b890600d9606d2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736223105 2023-04-17 12:57:07,160 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f6ac3a7f4a334a8899550481502034df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1681736223817 2023-04-17 12:57:07,160 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c8de98149a5b40fc9ccd3d619ae1256f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:07,164 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:07,166 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:07,183 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:07,185 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:07,186 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:57:07,186 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:07,202 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#20 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:07,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:07,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:07,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:07,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:07,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:07,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:07,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:07,300 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/985a84565cef481980e4c02861c7e376 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/985a84565cef481980e4c02861c7e376 2023-04-17 12:57:07,313 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 985a84565cef481980e4c02861c7e376(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:07,314 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:07,314 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736227015; duration=0sec 2023-04-17 12:57:07,314 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:07,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736287341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736287354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736287356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736287358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736287392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,394 DEBUG [Thread-148] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4302 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:07,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736287450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,568 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/66c5216bd781477198d9502ebb8a9721 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/66c5216bd781477198d9502ebb8a9721 2023-04-17 12:57:07,580 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 66c5216bd781477198d9502ebb8a9721(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:07,580 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:07,580 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736227014; duration=0sec 2023-04-17 12:57:07,580 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:07,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736287654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,745 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/de6c4bf765c54da88243080ca5d9a8a8 2023-04-17 12:57:07,796 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/90bea4f7f03d4b2e8aea8a9165700e81 2023-04-17 12:57:07,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:07,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736287977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:07,993 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/aea849aed3c148bdb99d4edcb61481d0 2023-04-17 12:57:08,006 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/de6c4bf765c54da88243080ca5d9a8a8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8 2023-04-17 12:57:08,018 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 12:57:08,021 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/90bea4f7f03d4b2e8aea8a9165700e81 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81 2023-04-17 12:57:08,077 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 12:57:08,092 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/aea849aed3c148bdb99d4edcb61481d0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0 2023-04-17 12:57:08,102 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 12:57:08,113 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for f14a4c6d8659d8cf4158b082301a3bb3 in 928ms, sequenceid=115, compaction requested=false 2023-04-17 12:57:08,113 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:08,114 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:08,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:08,119 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:08,119 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,119 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:08,119 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:08,121 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:08,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:08,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad745f6[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:08,128 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:08,128 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,129 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:08,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:08,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:08,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:08,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:08,132 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:08,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:08,157 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:08,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:08,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:08,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:08,157 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:08,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:08,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,161 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,162 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,165 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:08,165 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:08,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:08,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:08,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,176 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,176 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:08,176 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,177 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:08,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:08,177 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:08,177 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:08,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:08,278 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:08,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:08,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:08,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:08,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:08,306 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:08,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:08,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:08,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:08,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:08,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 12:57:08,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:08,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:08,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:08,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:08,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:08,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:08,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:08,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:08,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:08,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:08,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:08,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:08,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:08,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:08,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:08,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:08,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:08,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:08,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:08,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:08,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:08,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:08,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:08,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:08,505 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,505 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,506 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,506 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:08,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64fef481[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:08,509 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:08,509 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,510 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:08,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:08,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:08,511 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,511 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:08,512 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:08,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:08,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:08,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:08,515 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:08,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:08,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:08,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:08,515 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:08,518 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,518 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:08,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,519 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,527 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:08,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:08,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:08,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:08,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:08,535 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:08,535 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:08,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:08,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/0f5e0f6e208d4b918f50ac51dfdb045a 2023-04-17 12:57:08,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736288547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736288548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736288556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736288576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/c3963d6b3bed4b15b5202f89559af3b2 2023-04-17 12:57:08,635 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:08,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:08,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:08,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:08,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:08,680 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:08,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:08,684 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:08,684 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:08,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:08,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:08,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736288681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:08,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:08,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:08,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:08,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736288681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736288683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736288684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:08,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:08,705 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:08,705 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:08,847 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:08,847 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:08,848 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:08,848 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:08,848 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:08,848 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:08,848 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:08,848 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:08,848 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:08,849 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,849 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:08,849 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736288890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736288896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736288897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:08,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736288902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:08,959 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:08,959 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:08,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:08,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688d1905[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:08,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:08,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,059 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:09,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:09,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:09,061 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,061 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:09,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/694fa975113c43b68b2c64c50833e583 2023-04-17 12:57:09,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/0f5e0f6e208d4b918f50ac51dfdb045a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a 2023-04-17 12:57:09,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:09,115 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:09,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:09,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:09,121 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:09,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:09,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:09,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:09,121 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:09,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 12:57:09,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/c3963d6b3bed4b15b5202f89559af3b2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2 2023-04-17 12:57:09,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:09,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:09,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:09,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,173 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736289200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736289343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736289344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736289346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 12:57:09,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/694fa975113c43b68b2c64c50833e583 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583 2023-04-17 12:57:09,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:09,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:09,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:09,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:09,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:09,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:09,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:09,362 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:09,362 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:09,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:09,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583, entries=150, sequenceid=134, filesize=11.8 K 2023-04-17 12:57:09,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for f14a4c6d8659d8cf4158b082301a3bb3 in 897ms, sequenceid=134, compaction requested=true 2023-04-17 12:57:09,364 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:09,364 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:09,364 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:09,365 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:09,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:09,367 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:09,368 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:09,368 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:09,368 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:09,368 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:09,368 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:09,368 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/66c5216bd781477198d9502ebb8a9721, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.2 K 2023-04-17 12:57:09,368 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/a642c39bfc1544d7ac24dacf0bdae8c8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.2 K 2023-04-17 12:57:09,369 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 66c5216bd781477198d9502ebb8a9721, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:09,374 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a642c39bfc1544d7ac24dacf0bdae8c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:09,375 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting de6c4bf765c54da88243080ca5d9a8a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681736226073 2023-04-17 12:57:09,376 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 90bea4f7f03d4b2e8aea8a9165700e81, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681736226073 2023-04-17 12:57:09,376 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 0f5e0f6e208d4b918f50ac51dfdb045a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:09,377 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c3963d6b3bed4b15b5202f89559af3b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:09,425 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:09,438 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:09,472 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:09,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:09,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:09,482 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:09,482 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:09,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:09,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:09,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:09,488 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:09,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:09,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:09,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:09,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:09,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:09,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:09,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:09,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:09,491 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:09,491 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/cf1d2d88b24d4433a85ec0f9fe7790c9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/cf1d2d88b24d4433a85ec0f9fe7790c9 2023-04-17 12:57:09,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:09,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:09,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:09,498 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:09,498 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:09,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:09,507 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into cf1d2d88b24d4433a85ec0f9fe7790c9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:09,507 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:09,508 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736229364; duration=0sec 2023-04-17 12:57:09,508 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:09,508 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:09,514 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:09,514 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:09,514 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:09,514 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/985a84565cef481980e4c02861c7e376, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.2 K 2023-04-17 12:57:09,515 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 985a84565cef481980e4c02861c7e376, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681736225348 2023-04-17 12:57:09,515 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting aea849aed3c148bdb99d4edcb61481d0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681736226073 2023-04-17 12:57:09,516 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 694fa975113c43b68b2c64c50833e583, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:09,542 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:09,543 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/ab30b01260a84faf9ab515e631af38be 2023-04-17 12:57:09,589 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/135c0ae86ecd4ec7ad5e291780fc1b5c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/135c0ae86ecd4ec7ad5e291780fc1b5c 2023-04-17 12:57:09,604 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 135c0ae86ecd4ec7ad5e291780fc1b5c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:09,604 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:09,604 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736229367; duration=0sec 2023-04-17 12:57:09,604 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:09,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:09,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:09,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736289761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736289847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736289847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736289851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736289863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:09,892 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/34b580a9c36942859844329ef2b5a941 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/34b580a9c36942859844329ef2b5a941 2023-04-17 12:57:09,903 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 34b580a9c36942859844329ef2b5a941(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:09,903 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:09,904 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736229364; duration=0sec 2023-04-17 12:57:09,904 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:10,006 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8446adde36204f3ebe1286876d13a845 2023-04-17 12:57:10,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:10,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736290068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:10,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736290382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,420 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/e993c0f9217e41a0a34177eaba2217cc 2023-04-17 12:57:10,432 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/ab30b01260a84faf9ab515e631af38be as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be 2023-04-17 12:57:10,515 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be, entries=150, sequenceid=150, filesize=11.8 K 2023-04-17 12:57:10,517 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8446adde36204f3ebe1286876d13a845 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845 2023-04-17 12:57:10,523 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845, entries=150, sequenceid=150, filesize=11.8 K 2023-04-17 12:57:10,525 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/e993c0f9217e41a0a34177eaba2217cc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc 2023-04-17 12:57:10,535 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc, entries=150, sequenceid=150, filesize=11.8 K 2023-04-17 12:57:10,536 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f14a4c6d8659d8cf4158b082301a3bb3 in 1038ms, sequenceid=150, compaction requested=false 2023-04-17 12:57:10,536 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:10,536 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:10,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:10,596 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,596 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:10,597 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,597 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:10,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:10,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0be6ea[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:10,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:10,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,622 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,622 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:10,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:10,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:10,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:10,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:10,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:10,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:10,625 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:10,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:10,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:10,640 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:10,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:10,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:10,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:10,640 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:10,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:10,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:10,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:10,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:10,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,755 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:10,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:10,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:10,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:10,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:10,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:10,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:10,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:10,759 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:10,759 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:10,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:10,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:10,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:10,859 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:10,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:10,989 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 122 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:10,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:10,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:11,002 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:11,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,004 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:11,004 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:11,004 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736291000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736291002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736291005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736291006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,079 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/e369d1c590ec424a86e49776c4f85adb 2023-04-17 12:57:11,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:11,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:11,084 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:11,084 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:11,088 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:11,088 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:11,088 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,088 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:11,089 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:11,089 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:11,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:11,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:11,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:11,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:11,096 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,096 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,097 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,097 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:11,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e43fc28[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:11,101 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:11,101 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,105 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,106 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:11,107 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:11,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:11,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:11,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736291111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736291115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736291116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736291115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:11,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:11,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:11,143 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:11,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:11,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:11,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:11,143 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:11,153 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,153 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,153 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,154 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,156 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,169 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:11,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:11,170 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:11,170 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,270 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:11,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:11,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:11,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:11,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:11,283 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:11,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,284 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:11,284 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:11,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,289 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,289 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,290 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,290 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:11,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:11,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:11,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:11,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:11,294 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:11,294 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:11,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:11,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:11,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:11,296 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:11,296 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:11,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:11,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:11,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:11,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,297 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:11,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8a5b9d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:11,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:11,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,309 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:11,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:11,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:11,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:11,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:11,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,312 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736291321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736291323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:11,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:11,326 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:11,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:11,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:11,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:11,326 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:11,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,331 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736291326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736291326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:11,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:11,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:11,336 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:11,337 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:11,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736291397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,399 DEBUG [Thread-148] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8307 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:11,437 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:11,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:11,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:11,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:11,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:11,445 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:11,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:11,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:11,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,453 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:11,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:11,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:11,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:11,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:11,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:11,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:11,471 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,471 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:11,471 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:11,471 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:11,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:11,473 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:11,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d4f3c18[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:11,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:11,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:11,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:11,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:11,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:11,480 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:11,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:11,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:11,487 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:11,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:11,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:11,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:11,487 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:11,489 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,489 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,489 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,490 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,491 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,493 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,493 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:11,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:11,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,499 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,499 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,499 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:11,499 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:11,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/6eeefa963214457aa9fc0d5b807887ae 2023-04-17 12:57:11,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/a4a1343b4d194f9dab8c571718adf532 2023-04-17 12:57:11,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/e369d1c590ec424a86e49776c4f85adb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb 2023-04-17 12:57:11,599 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb, entries=200, sequenceid=170, filesize=14.1 K 2023-04-17 12:57:11,600 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:11,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:11,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/6eeefa963214457aa9fc0d5b807887ae as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae 2023-04-17 12:57:11,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:11,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:11,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:11,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:11,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:11,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:11,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,609 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:11,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:11,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:11,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:11,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:11,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:11,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:11,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:11,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:11,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:11,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:11,615 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 12:57:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,619 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,620 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,620 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:11,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c92b5ed[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:11,621 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:11,621 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,623 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:11,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:11,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:11,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/a4a1343b4d194f9dab8c571718adf532 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532 2023-04-17 12:57:11,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,627 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736291630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736291628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:11,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:11,634 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:11,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:11,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:11,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:11,635 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:11,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:11,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:11,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736291633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:11,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736291633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,643 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,643 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:11,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:11,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:11,644 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:11,647 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:11,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:11,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:11,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 12:57:11,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f14a4c6d8659d8cf4158b082301a3bb3 in 801ms, sequenceid=170, compaction requested=true 2023-04-17 12:57:11,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:11,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:11,655 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:11,655 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:11,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:11,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:11,659 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:11,659 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:11,659 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,659 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/34b580a9c36942859844329ef2b5a941, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=37.9 K 2023-04-17 12:57:11,659 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:11,660 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:11,660 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,660 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/cf1d2d88b24d4433a85ec0f9fe7790c9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.6 K 2023-04-17 12:57:11,660 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 34b580a9c36942859844329ef2b5a941, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:11,667 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ab30b01260a84faf9ab515e631af38be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736228483 2023-04-17 12:57:11,667 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting cf1d2d88b24d4433a85ec0f9fe7790c9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:11,668 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e369d1c590ec424a86e49776c4f85adb, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:11,668 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8446adde36204f3ebe1286876d13a845, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736228483 2023-04-17 12:57:11,669 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6eeefa963214457aa9fc0d5b807887ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:11,680 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:11,687 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:11,747 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:11,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:11,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:11,754 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/71332b7be7224521ba74491b3b443498 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/71332b7be7224521ba74491b3b443498 2023-04-17 12:57:11,759 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/6eb355024f154aea8b39341e29c9808a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eb355024f154aea8b39341e29c9808a 2023-04-17 12:57:11,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:11,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:11,770 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:11,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:11,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:11,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,771 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 6eb355024f154aea8b39341e29c9808a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:11,771 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 71332b7be7224521ba74491b3b443498(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:11,771 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:11,771 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736231654; duration=0sec 2023-04-17 12:57:11,772 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:11,772 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:11,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,771 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:11,772 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:11,772 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736231655; duration=0sec 2023-04-17 12:57:11,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:11,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:11,773 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:11,772 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:11,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:11,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:11,774 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:11,774 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:11,774 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:11,774 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/135c0ae86ecd4ec7ad5e291780fc1b5c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.6 K 2023-04-17 12:57:11,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:11,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:11,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:11,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:11,775 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:11,775 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:11,775 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:11,775 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 135c0ae86ecd4ec7ad5e291780fc1b5c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681736228366 2023-04-17 12:57:11,776 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e993c0f9217e41a0a34177eaba2217cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736228483 2023-04-17 12:57:11,776 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a4a1343b4d194f9dab8c571718adf532, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:11,790 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:11,805 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:11,805 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:11,805 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:11,805 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:11,806 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 12:57:11,806 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:11,806 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:11,806 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:11,806 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:11,806 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:11,807 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:11,840 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/8dd415abd5ba49889c062c4ec2df674d 2023-04-17 12:57:11,846 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/36ed33d3bb634778ba6997c0bd11718a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/36ed33d3bb634778ba6997c0bd11718a 2023-04-17 12:57:11,863 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 36ed33d3bb634778ba6997c0bd11718a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:11,863 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:11,863 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736231656; duration=0sec 2023-04-17 12:57:11,863 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:11,884 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/530a5949f86345f19d80a33b2c0ee4d0 2023-04-17 12:57:11,933 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/0261be9ef9e9407aaf4394586f923eb2 2023-04-17 12:57:11,946 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/8dd415abd5ba49889c062c4ec2df674d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d 2023-04-17 12:57:11,960 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d, entries=150, sequenceid=187, filesize=11.8 K 2023-04-17 12:57:11,961 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/530a5949f86345f19d80a33b2c0ee4d0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0 2023-04-17 12:57:12,045 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0, entries=150, sequenceid=187, filesize=11.8 K 2023-04-17 12:57:12,047 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/0261be9ef9e9407aaf4394586f923eb2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2 2023-04-17 12:57:12,057 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2, entries=150, sequenceid=187, filesize=11.8 K 2023-04-17 12:57:12,058 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for f14a4c6d8659d8cf4158b082301a3bb3 in 252ms, sequenceid=187, compaction requested=false 2023-04-17 12:57:12,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:12,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:12,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:12,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:12,059 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:12,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:12,099 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,099 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,099 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:12,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d21b4eb[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:12,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:12,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,109 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,109 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,109 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:12,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:12,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:12,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:12,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:12,195 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:12,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:12,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:12,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:12,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:12,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:12,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:12,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:12,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:12,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:12,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:12,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:12,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:12,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:12,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:12,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:12,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:12,209 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:12,209 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:12,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:12,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/d1b21f1afa0c487ab5a2653a47add366 2023-04-17 12:57:12,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736232227, max timestamp=1681736232241Num uniques 859; 2023-04-17 12:57:12,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:12,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736232227, max timestamp=1681736232240Num uniques 859; 2023-04-17 12:57:12,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:12,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/3164f4ed71d74075a2578e3c43465d51 2023-04-17 12:57:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736232227, max timestamp=1681736232260Num uniques 859; 2023-04-17 12:57:12,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:57:12,309 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:12,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:12,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:12,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:12,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:12,327 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:12,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:12,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:12,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:12,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:12,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:12,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:12,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:12,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:12,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:12,338 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:12,338 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:12,338 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:12,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:12,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:12,459 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:12,459 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:12,459 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:12,459 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:12,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:12,460 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:12,460 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,460 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:12,460 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:12,461 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 122 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:12,461 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 119 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:12,461 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 113 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:12,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736232309, max timestamp=1681736232347Num uniques 1018; 2023-04-17 12:57:12,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736292338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:57:12,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736232334, max timestamp=1681736232347Num uniques 1009; 2023-04-17 12:57:12,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-17 12:57:12,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736232309, max timestamp=1681736232347Num uniques 1018; 2023-04-17 12:57:12,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:57:12,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736292503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736292504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,508 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,508 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,509 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,509 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:12,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22b84c61[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:12,510 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:12,510 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736292507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:12,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:12,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:12,516 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,516 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:12,547 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:12,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:12,551 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:12,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:12,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:12,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:12,551 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:12,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:12,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,556 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,558 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,561 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:12,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:12,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:12,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:12,562 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:12,562 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:12,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:12,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:12,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:12,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736292570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736292607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736292609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33660 deadline: 1681736292612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,662 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:12,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:12,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:12,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:12,668 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,690 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:12,690 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:12,691 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,691 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:12,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:12,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:12,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:12,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:12,692 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:12,692 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:12,692 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:12,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:12,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:12,717 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:12,717 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:12,717 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:12,717 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:12,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:12,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:12,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:12,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:12,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736292774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,779 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,779 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:12,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:12,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:12,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46731bd2[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:12,784 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:12,784 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736292812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736292812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:12,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736292819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/ce06a90d2be347208d3a7e21eb74c97b 2023-04-17 12:57:12,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/d1b21f1afa0c487ab5a2653a47add366 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366 2023-04-17 12:57:12,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366, entries=150, sequenceid=199, filesize=11.8 K 2023-04-17 12:57:12,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/3164f4ed71d74075a2578e3c43465d51 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51 2023-04-17 12:57:12,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:12,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:12,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:12,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:12,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:12,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:12,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:12,922 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:12,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:12,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:12,923 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:12,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:12,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:12,924 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:12,924 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:12,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51, entries=150, sequenceid=199, filesize=11.8 K 2023-04-17 12:57:12,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/ce06a90d2be347208d3a7e21eb74c97b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b 2023-04-17 12:57:13,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b, entries=150, sequenceid=199, filesize=11.8 K 2023-04-17 12:57:13,015 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-17 12:57:13,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=135.62 KB/138872 for f14a4c6d8659d8cf4158b082301a3bb3 in 821ms, sequenceid=199, compaction requested=true 2023-04-17 12:57:13,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:13,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:13,016 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:13,016 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:13,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:13,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:13,018 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:13,018 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:13,018 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:13,018 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,018 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:13,018 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/71332b7be7224521ba74491b3b443498, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.7 K 2023-04-17 12:57:13,018 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,018 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eb355024f154aea8b39341e29c9808a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.7 K 2023-04-17 12:57:13,019 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 71332b7be7224521ba74491b3b443498, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:13,019 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6eb355024f154aea8b39341e29c9808a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:13,020 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 530a5949f86345f19d80a33b2c0ee4d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736230997 2023-04-17 12:57:13,020 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8dd415abd5ba49889c062c4ec2df674d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736230997 2023-04-17 12:57:13,020 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3164f4ed71d74075a2578e3c43465d51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:13,020 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d1b21f1afa0c487ab5a2653a47add366, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:13,032 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:13,033 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:13,055 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,055 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,055 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:13,058 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,076 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/475d95c20d7c4f4d844aef6e75f1b428 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/475d95c20d7c4f4d844aef6e75f1b428 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:13,079 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:13,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:13,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,080 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:13,080 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:13,079 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:13,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:13,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:13,083 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/be33ade6dfc14366bd534e2b38a40229 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/be33ade6dfc14366bd534e2b38a40229 2023-04-17 12:57:13,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:13,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=142.33 KB heapSize=402.63 KB 2023-04-17 12:57:13,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:13,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:13,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:13,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:13,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:13,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:13,106 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 475d95c20d7c4f4d844aef6e75f1b428(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:13,107 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:13,107 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736233016; duration=0sec 2023-04-17 12:57:13,107 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:13,107 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:13,113 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:13,113 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:13,113 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,113 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/36ed33d3bb634778ba6997c0bd11718a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=35.7 K 2023-04-17 12:57:13,114 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 36ed33d3bb634778ba6997c0bd11718a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736229743 2023-04-17 12:57:13,115 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into be33ade6dfc14366bd534e2b38a40229(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:13,115 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:13,115 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736233016; duration=0sec 2023-04-17 12:57:13,115 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:13,116 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 0261be9ef9e9407aaf4394586f923eb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736230997 2023-04-17 12:57:13,116 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ce06a90d2be347208d3a7e21eb74c97b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:13,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736293135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736293137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736293143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736293145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/cddcc5780eac4cde9ac290d05989c9de 2023-04-17 12:57:13,180 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:13,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:13,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:13,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:13,192 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:13,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:13,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:13,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:13,193 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,193 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,194 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:13,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:13,195 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,195 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,195 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:13,199 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/eaf6dd6359a74dc5a540ce3b9c61c154 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/eaf6dd6359a74dc5a540ce3b9c61c154 2023-04-17 12:57:13,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:13,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:13,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:13,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:13,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:13,214 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into eaf6dd6359a74dc5a540ce3b9c61c154(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:13,214 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:13,214 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736233017; duration=0sec 2023-04-17 12:57:13,214 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:13,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:13,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:13,226 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,226 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:13,226 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:13,226 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:13,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:13,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/9711dd513d6840a389519fcf8d2cd7f9 2023-04-17 12:57:13,246 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:13,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:13,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,258 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:13,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:13,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:13,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f27f32c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:13,260 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:13,260 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,261 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,261 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:13,262 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:13,262 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:13,263 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:13,263 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,263 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:13,263 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:13,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:13,263 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:13,265 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,265 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,265 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:13,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:13,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:13,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:13,265 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:13,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,267 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,271 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,271 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,271 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,271 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:13,271 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:13,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:13,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:13,272 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,272 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:13,272 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,272 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:13,272 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:13,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736293245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:13,276 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:13,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736293245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:13,276 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:13,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:13,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736293258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:13,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736293258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.93 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/90f22b9f4cb44244b9e8452112862956 2023-04-17 12:57:13,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/cddcc5780eac4cde9ac290d05989c9de as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de 2023-04-17 12:57:13,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:57:13,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/9711dd513d6840a389519fcf8d2cd7f9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9 2023-04-17 12:57:13,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:57:13,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/90f22b9f4cb44244b9e8452112862956 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956 2023-04-17 12:57:13,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:57:13,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~142.33 KB/145742, heapSize ~402.58 KB/412240, currentSize=46.96 KB/48090 for f14a4c6d8659d8cf4158b082301a3bb3 in 264ms, sequenceid=253, compaction requested=false 2023-04-17 12:57:13,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:13,372 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:13,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:13,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:13,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:13,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:13,378 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:13,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:13,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:13,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:13,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,379 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:13,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:13,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:13,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:13,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:13,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:13,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:13,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:13,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:13,397 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:13,397 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:13,397 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:13,397 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:13,398 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:57:13,399 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:13,399 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:13,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:13,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:13,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:13,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:13,420 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/7e7d657caf97449bab82693226a398c8 2023-04-17 12:57:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:13,483 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:13,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736293523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736233513, max timestamp=1681736233530Num uniques 859; 2023-04-17 12:57:13,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736233513, max timestamp=1681736233532Num uniques 859; 2023-04-17 12:57:13,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:13,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:13,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736233513, max timestamp=1681736233520Num uniques 859; 2023-04-17 12:57:13,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:57:13,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736233581, max timestamp=1681736233618Num uniques 1009; 2023-04-17 12:57:13,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-17 12:57:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736233581, max timestamp=1681736233618Num uniques 1018; 2023-04-17 12:57:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736233581, max timestamp=1681736233618Num uniques 1018; 2023-04-17 12:57:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:57:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:57:13,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736293657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736293658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736293664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736293669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736293766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736293769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736293775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33624 deadline: 1681736293775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:13,886 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/96ecb945c83244a28251de048b9dd290 2023-04-17 12:57:14,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736294008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736294009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736294009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736294009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,051 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/a3fe340b8c1a4e1fb76765ae2e3fda88 2023-04-17 12:57:14,057 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/7e7d657caf97449bab82693226a398c8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8 2023-04-17 12:57:14,068 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8, entries=150, sequenceid=266, filesize=11.9 K 2023-04-17 12:57:14,069 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/96ecb945c83244a28251de048b9dd290 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290 2023-04-17 12:57:14,077 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290, entries=150, sequenceid=266, filesize=11.9 K 2023-04-17 12:57:14,079 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/a3fe340b8c1a4e1fb76765ae2e3fda88 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88 2023-04-17 12:57:14,087 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88, entries=150, sequenceid=266, filesize=11.9 K 2023-04-17 12:57:14,089 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f14a4c6d8659d8cf4158b082301a3bb3 in 691ms, sequenceid=266, compaction requested=true 2023-04-17 12:57:14,089 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:14,089 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:14,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:14,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:14,089 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:14,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:14,097 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,097 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,097 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,097 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:14,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f419ccf[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:14,100 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:14,100 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,232 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:14,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:14,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:14,238 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,238 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,238 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:14,239 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:14,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:14,243 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:14,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:14,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:14,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:14,243 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:14,248 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,248 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,248 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,251 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,254 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:14,254 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:14,254 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:14,254 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:14,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:14,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:14,254 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,255 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,255 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,255 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,255 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:14,255 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:14,255 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:14,255 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:14,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-17 12:57:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:14,336 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:14,337 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:14,337 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:14,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33662 deadline: 1681736294347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736294347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33666 deadline: 1681736294351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,355 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:14,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:14,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:14,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:14,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:14,358 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:14,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:14,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:14,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:14,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736294452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736294453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736294453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736294533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,543 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,544 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,544 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:14,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:14,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:14,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:14,545 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:14,545 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:14,545 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:14,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:14,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:14,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:14,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:14,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:14,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:14,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:14,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:14,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,555 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:14,555 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:14,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/9ced730983eb49a5b83d09d131ea9f63 2023-04-17 12:57:14,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,592 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,592 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,592 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:14,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7a1a02[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:14,602 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:14,602 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:14,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:14,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:14,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:14,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,640 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,640 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:14,640 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:14,642 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:14,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:14,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:14,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:14,642 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:14,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736294655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736294656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736294657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:14,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:14,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:14,781 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,784 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,919 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,919 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,919 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:14,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:14,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:14,919 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:14,919 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:14,919 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:14,919 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:14,920 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:14,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:14,919 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,920 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,920 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:14,920 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:14,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:14,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:14,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736294959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736294959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:14,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736294960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,020 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:15,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:15,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:15,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:15,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:15,024 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:15,024 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,025 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:15,025 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:15,025 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,025 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,037 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:15,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:15,037 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:15,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:15,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:15,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:15,039 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,039 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:15,039 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:15,039 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:15,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:15,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:15,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:15,043 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,043 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,043 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:15,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:15,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@710939b4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:15,047 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:15,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:15,047 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,047 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:15,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:15,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:15,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:15,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:15,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:15,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736295050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:15,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:15,055 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:15,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:15,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:15,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:15,056 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:15,057 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,057 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:15,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,058 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,064 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:15,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:15,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:15,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,069 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:15,069 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:15,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:15,069 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:15,169 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:15,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:15,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:15,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:15,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:15,172 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:15,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:15,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:15,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,172 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/ae0e72a883c545cdbeb835b15a154fe7 2023-04-17 12:57:15,357 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:15,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:15,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:15,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:15,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:15,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:15,359 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:15,359 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,359 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:15,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:15,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:15,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:15,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:15,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:15,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:15,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a70a2a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:15,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:15,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736295464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:15,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736295466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,469 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:15,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:15,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:15,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:15,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736295468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:15,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:15,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:15,478 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:15,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:15,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:15,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:15,478 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:15,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:15,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:15,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:15,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:15,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:15,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:15,584 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:15,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:15,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:15,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:15,584 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:15,584 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:15,584 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:15,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:15,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/b4cbffe7f12c4472abf9f1c3a6e2decf 2023-04-17 12:57:15,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/9ced730983eb49a5b83d09d131ea9f63 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63 2023-04-17 12:57:15,684 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:15,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:15,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:15,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:15,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:15,752 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:15,752 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,752 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:15,752 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:15,753 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,753 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63, entries=200, sequenceid=319, filesize=14.3 K 2023-04-17 12:57:15,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/ae0e72a883c545cdbeb835b15a154fe7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7 2023-04-17 12:57:15,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7, entries=150, sequenceid=319, filesize=11.9 K 2023-04-17 12:57:15,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/b4cbffe7f12c4472abf9f1c3a6e2decf as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf 2023-04-17 12:57:15,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf, entries=150, sequenceid=319, filesize=11.9 K 2023-04-17 12:57:15,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for f14a4c6d8659d8cf4158b082301a3bb3 in 1448ms, sequenceid=319, compaction requested=true 2023-04-17 12:57:15,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:15,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:15,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:15,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:15,784 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:15,784 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51396 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:15,788 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,788 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,788 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/475d95c20d7c4f4d844aef6e75f1b428, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=47.8 K 2023-04-17 12:57:15,788 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/be33ade6dfc14366bd534e2b38a40229, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=50.2 K 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting be33ade6dfc14366bd534e2b38a40229, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:15,788 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 475d95c20d7c4f4d844aef6e75f1b428, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:15,789 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting cddcc5780eac4cde9ac290d05989c9de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736232493 2023-04-17 12:57:15,789 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9711dd513d6840a389519fcf8d2cd7f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736232493 2023-04-17 12:57:15,789 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7e7d657caf97449bab82693226a398c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681736233127 2023-04-17 12:57:15,790 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 96ecb945c83244a28251de048b9dd290, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681736233127 2023-04-17 12:57:15,790 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 9ced730983eb49a5b83d09d131ea9f63, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:15,790 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ae0e72a883c545cdbeb835b15a154fe7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:15,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:15,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:15,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:15,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:15,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:15,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:15,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:15,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:15,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:15,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:15,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:15,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:15,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:15,814 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 12:57:15,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:15,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:15,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:15,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:15,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:15,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:15,848 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:15,848 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:15,887 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/5b06c8b9ff924194b5bf716027afec25 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/5b06c8b9ff924194b5bf716027afec25 2023-04-17 12:57:15,889 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/8d6ebc05da35423b92dc22986eaae53a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8d6ebc05da35423b92dc22986eaae53a 2023-04-17 12:57:15,904 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 5b06c8b9ff924194b5bf716027afec25(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:15,904 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:15,904 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=12, startTime=1681736235784; duration=0sec 2023-04-17 12:57:15,905 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:15,905 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:15,906 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 8d6ebc05da35423b92dc22986eaae53a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:15,906 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:15,906 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=12, startTime=1681736235784; duration=0sec 2023-04-17 12:57:15,907 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:15,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:15,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:15,908 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:15,908 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/eaf6dd6359a74dc5a540ce3b9c61c154, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=47.8 K 2023-04-17 12:57:15,909 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting eaf6dd6359a74dc5a540ce3b9c61c154, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681736232191 2023-04-17 12:57:15,910 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 90f22b9f4cb44244b9e8452112862956, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736232493 2023-04-17 12:57:15,911 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a3fe340b8c1a4e1fb76765ae2e3fda88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681736233127 2023-04-17 12:57:15,911 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b4cbffe7f12c4472abf9f1c3a6e2decf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:15,931 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/24559f6ce54c4423a32a115a728e8317 2023-04-17 12:57:15,946 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:16,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:16,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:16,096 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/53ed4036154f4356b55a4389ade6f493 2023-04-17 12:57:16,114 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/97bfefb739514e7091a017c20863e59e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/97bfefb739514e7091a017c20863e59e 2023-04-17 12:57:16,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736236125, max timestamp=1681736236141Num uniques 859; 2023-04-17 12:57:16,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736236125, max timestamp=1681736236141Num uniques 859; 2023-04-17 12:57:16,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:16,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736236125, max timestamp=1681736236141Num uniques 859; 2023-04-17 12:57:16,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:16,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:16,182 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 97bfefb739514e7091a017c20863e59e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:16,182 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:16,182 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=12, startTime=1681736235784; duration=0sec 2023-04-17 12:57:16,183 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:16,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736236166, max timestamp=1681736236303Num uniques 1018; 2023-04-17 12:57:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736236166, max timestamp=1681736236303Num uniques 1018; 2023-04-17 12:57:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:57:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:57:16,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736236166, max timestamp=1681736236303Num uniques 1018; 2023-04-17 12:57:16,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:57:16,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681736236351, max timestamp=1681736236369Num uniques 1027; 2023-04-17 12:57:16,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 12:57:16,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681736236351, max timestamp=1681736236369Num uniques 1027; 2023-04-17 12:57:16,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 12:57:16,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681736236351, max timestamp=1681736236369Num uniques 1027; 2023-04-17 12:57:16,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-17 12:57:16,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736296489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8496, totalHeapSize=19584, min timestamp=1681736236480, max timestamp=1681736236497Num uniques 1036; 2023-04-17 12:57:16,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=47426, new segment data size=8496, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 12:57:16,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8500, totalHeapSize=19584, min timestamp=1681736236480, max timestamp=1681736236497Num uniques 1036; 2023-04-17 12:57:16,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47430, new segment data size=8500, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 12:57:16,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8504, totalHeapSize=19584, min timestamp=1681736236480, max timestamp=1681736236497Num uniques 1036; 2023-04-17 12:57:16,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47434, new segment data size=8504, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-17 12:57:16,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736296569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681736236558, max timestamp=1681736236572Num uniques 1045; 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681736236558, max timestamp=1681736236572Num uniques 1045; 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681736236558, max timestamp=1681736236572Num uniques 1045; 2023-04-17 12:57:16,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=18090, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-04-17 12:57:16,591 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/5ad8e4bd58744f20997c3458aee08440 2023-04-17 12:57:16,599 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/24559f6ce54c4423a32a115a728e8317 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317 2023-04-17 12:57:16,605 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317, entries=150, sequenceid=328, filesize=11.9 K 2023-04-17 12:57:16,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/53ed4036154f4356b55a4389ade6f493 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493 2023-04-17 12:57:16,619 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493, entries=150, sequenceid=328, filesize=11.9 K 2023-04-17 12:57:16,620 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/5ad8e4bd58744f20997c3458aee08440 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440 2023-04-17 12:57:16,634 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440, entries=150, sequenceid=328, filesize=11.9 K 2023-04-17 12:57:16,635 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for f14a4c6d8659d8cf4158b082301a3bb3 in 821ms, sequenceid=328, compaction requested=false 2023-04-17 12:57:16,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:16,636 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:16,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:16,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:16,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:16,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:16,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:16,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=147.60 KB heapSize=402.70 KB 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:16,636 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:16,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:16,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:16,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:16,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:16,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:16,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c35535[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:16,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:16,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:16,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:16,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:16,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:16,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:16,643 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:16,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:16,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:16,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:16,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:16,647 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:16,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:16,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:16,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:16,647 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:16,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:16,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:16,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:16,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:16,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,651 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:16,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/69a8825a771f44faa4c0e7591d3cdac9 2023-04-17 12:57:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:16,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:16,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,655 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:16,655 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:16,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:16,655 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:16,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:16,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:16,655 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:16,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:16,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736296655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736296657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736296665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8651d0b9dd104bac9bd6b8a16ea111df 2023-04-17 12:57:16,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736296677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,755 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:16,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:16,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:16,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736296763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736296770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:16,787 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:16,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:16,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:16,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:16,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,787 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:16,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:16,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736296786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/6ac98a6927924d7ea713ea6f4511f46a 2023-04-17 12:57:16,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/69a8825a771f44faa4c0e7591d3cdac9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9 2023-04-17 12:57:16,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9, entries=250, sequenceid=434, filesize=16.7 K 2023-04-17 12:57:16,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8651d0b9dd104bac9bd6b8a16ea111df as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df 2023-04-17 12:57:16,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df, entries=150, sequenceid=434, filesize=11.9 K 2023-04-17 12:57:16,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/6ac98a6927924d7ea713ea6f4511f46a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a 2023-04-17 12:57:16,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a, entries=150, sequenceid=434, filesize=11.9 K 2023-04-17 12:57:16,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~402.66 KB/412320, currentSize=40.25 KB/41220 for f14a4c6d8659d8cf4158b082301a3bb3 in 188ms, sequenceid=434, compaction requested=true 2023-04-17 12:57:16,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:16,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:16,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:16,825 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:16,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:16,825 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:16,826 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:16,826 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42086 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:16,826 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:16,826 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:16,826 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:16,826 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:16,826 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8d6ebc05da35423b92dc22986eaae53a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=41.1 K 2023-04-17 12:57:16,826 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/5b06c8b9ff924194b5bf716027afec25, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.3 K 2023-04-17 12:57:16,827 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8d6ebc05da35423b92dc22986eaae53a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:16,827 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5b06c8b9ff924194b5bf716027afec25, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:16,827 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 24559f6ce54c4423a32a115a728e8317, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681736234337 2023-04-17 12:57:16,827 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 53ed4036154f4356b55a4389ade6f493, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681736234337 2023-04-17 12:57:16,828 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 69a8825a771f44faa4c0e7591d3cdac9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:16,828 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8651d0b9dd104bac9bd6b8a16ea111df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:16,865 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:16,867 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:16,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:16,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:16,922 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:16,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:16,922 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:16,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:16,923 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:16,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:16,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:16,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:16,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:16,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:16,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:16,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:16,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:16,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:16,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:16,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:16,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:16,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:16,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:16,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:16,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:16,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736236990, max timestamp=1681736237023Num uniques 859; 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736236990, max timestamp=1681736237023Num uniques 859; 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736236990, max timestamp=1681736237023Num uniques 859; 2023-04-17 12:57:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:17,025 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/7d8e0c43ae024c88b10c1e5423005e03 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/7d8e0c43ae024c88b10c1e5423005e03 2023-04-17 12:57:17,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,028 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:17,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d96f165[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:17,031 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:17,031 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/2ebd227094ce4183a414bef73a5b4646 2023-04-17 12:57:17,036 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 7d8e0c43ae024c88b10c1e5423005e03(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:17,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,037 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:17,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,037 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:17,037 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:17,037 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:17,037 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736236825; duration=0sec 2023-04-17 12:57:17,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:17,039 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:17,040 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:17,043 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:17,043 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:17,043 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:17,043 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/97bfefb739514e7091a017c20863e59e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.3 K 2023-04-17 12:57:17,044 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 97bfefb739514e7091a017c20863e59e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681736233635 2023-04-17 12:57:17,045 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5ad8e4bd58744f20997c3458aee08440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681736234337 2023-04-17 12:57:17,045 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6ac98a6927924d7ea713ea6f4511f46a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:17,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:17,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,077 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:17,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:17,079 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:17,080 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:17,080 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:17,080 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:17,080 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:17,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736297088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736297087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736297093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736297093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,177 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,177 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:17,177 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,177 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736297194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736297194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736297198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736297198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:17,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:17,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:17,318 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:17,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:17,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:17,319 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:17,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:17,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,319 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:17,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:17,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736297397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736297397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736297401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736297404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,419 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:17,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:17,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:17,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:17,423 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:17,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:17,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:17,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:17,424 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,424 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,433 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/831d4ed6a7224b26ba8d2f6a539e8bbf as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/831d4ed6a7224b26ba8d2f6a539e8bbf 2023-04-17 12:57:17,434 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:17,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,434 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:17,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:17,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:17,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:17,439 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:17,439 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:17,439 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:17,439 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:17,439 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:17,439 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:17,439 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:17,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:17,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:17,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:17,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:17,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:17,450 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 831d4ed6a7224b26ba8d2f6a539e8bbf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:17,450 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:17,450 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736236825; duration=0sec 2023-04-17 12:57:17,451 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:17,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34389039[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:17,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:17,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:17,457 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:17,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:17,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:17,465 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:17,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:17,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:17,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:17,471 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:17,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:17,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:17,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:17,471 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:17,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,473 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:17,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:17,481 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:17,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:17,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,482 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,482 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,482 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,482 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:17,482 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:17,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:17,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:17,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/aa73b4e0a78945dea819a3d181220db4 2023-04-17 12:57:17,582 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:17,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:17,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:17,590 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:17,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:17,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:17,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:17,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,650 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/992b7ed41f1045a89d30a0e6535b947e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/992b7ed41f1045a89d30a0e6535b947e 2023-04-17 12:57:17,658 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 992b7ed41f1045a89d30a0e6535b947e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:17,658 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:17,658 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736236825; duration=0sec 2023-04-17 12:57:17,658 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:17,668 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:17,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:17,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:17,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:17,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:17,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:17,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:17,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:17,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:17,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:17,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:17,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:17,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:17,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:17,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:17,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:17,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65eab7fa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:17,692 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:17,692 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736297704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736297712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736297713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:17,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736297716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:17,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:17,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:17,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:17,844 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,844 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:17,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:17,936 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:17,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:17,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:17,942 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:17,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:17,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:17,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:17,942 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:18,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/90938baa8b6f4fe1b0e62338e4f40807 2023-04-17 12:57:18,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/2ebd227094ce4183a414bef73a5b4646 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646 2023-04-17 12:57:18,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646, entries=200, sequenceid=444, filesize=14.3 K 2023-04-17 12:57:18,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/aa73b4e0a78945dea819a3d181220db4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4 2023-04-17 12:57:18,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4, entries=150, sequenceid=444, filesize=11.9 K 2023-04-17 12:57:18,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/90938baa8b6f4fe1b0e62338e4f40807 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807 2023-04-17 12:57:18,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807, entries=150, sequenceid=444, filesize=11.9 K 2023-04-17 12:57:18,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for f14a4c6d8659d8cf4158b082301a3bb3 in 1225ms, sequenceid=444, compaction requested=false 2023-04-17 12:57:18,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:18,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:18,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:18,154 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:18,154 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:18,155 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:18,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:18,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:18,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:18,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:18,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736298222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736298223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736298225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736298225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,255 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:18,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:18,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:18,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:18,279 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:18,279 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:18,279 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:18,279 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:18,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736298326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736298329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736298329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,387 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:18,387 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:18,387 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,387 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:18,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:18,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:18,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:18,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:18,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:18,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:18,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:18,389 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,390 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:18,390 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:18,390 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:18,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:18,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:18,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:18,456 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:18,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc51737[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:18,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:18,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,461 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:18,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:18,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:18,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:18,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736298529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736298534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736298535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,539 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:18,539 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,539 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,539 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:18,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:18,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:18,543 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:18,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:18,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:18,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:18,543 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:18,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/98118140dec642069cf7af47f7648f8c 2023-04-17 12:57:18,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/9c37c280e5b24d0d96698d128beb1767 2023-04-17 12:57:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,661 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/5d0ac188e494417784e29bcdf6671707 2023-04-17 12:57:18,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/98118140dec642069cf7af47f7648f8c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c 2023-04-17 12:57:18,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c, entries=150, sequenceid=483, filesize=11.9 K 2023-04-17 12:57:18,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/9c37c280e5b24d0d96698d128beb1767 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767 2023-04-17 12:57:18,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767, entries=150, sequenceid=483, filesize=11.9 K 2023-04-17 12:57:18,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/5d0ac188e494417784e29bcdf6671707 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,781 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:18,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:18,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:18,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:18,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:18,782 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,782 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,782 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:18,782 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707, entries=150, sequenceid=483, filesize=11.9 K 2023-04-17 12:57:18,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for f14a4c6d8659d8cf4158b082301a3bb3 in 578ms, sequenceid=483, compaction requested=true 2023-04-17 12:57:18,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:18,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:18,791 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:18,791 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:18,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:18,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:18,793 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:18,793 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:18,793 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:18,793 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:18,793 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,793 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,793 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/7d8e0c43ae024c88b10c1e5423005e03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.4 K 2023-04-17 12:57:18,793 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/831d4ed6a7224b26ba8d2f6a539e8bbf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=38.8 K 2023-04-17 12:57:18,793 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7d8e0c43ae024c88b10c1e5423005e03, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:18,794 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 831d4ed6a7224b26ba8d2f6a539e8bbf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:18,794 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2ebd227094ce4183a414bef73a5b4646, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681736236642 2023-04-17 12:57:18,794 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting aa73b4e0a78945dea819a3d181220db4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681736236653 2023-04-17 12:57:18,794 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 98118140dec642069cf7af47f7648f8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:18,794 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9c37c280e5b24d0d96698d128beb1767, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:18,807 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:18,811 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:18,882 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:18,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:18,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:18,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:18,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:18,885 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:18,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:18,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:18,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:18,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:18,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:57:18,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:18,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:18,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:18,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:18,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:18,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:18,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,915 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:18,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:18,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:18,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:18,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:18,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:18,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:18,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:18,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:18,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:18,919 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,919 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:18,920 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:18,920 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:18,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:18,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:18,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:18,920 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe 2023-04-17 12:57:18,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,921 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:18,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4979767[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:18,924 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:18,924 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,925 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:18,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:18,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:18,926 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,926 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:18,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:18,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:18,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:18,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,936 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into b1ad62d7a1ad4a9eb1f8c3b461c043fe(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:18,936 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,936 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736238791; duration=0sec 2023-04-17 12:57:18,936 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:18,936 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:18,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:18,939 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:18,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:18,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:18,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:18,939 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:18,939 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:18,940 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:18,940 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:18,940 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/992b7ed41f1045a89d30a0e6535b947e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.4 K 2023-04-17 12:57:18,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,941 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 992b7ed41f1045a89d30a0e6535b947e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1681736236568 2023-04-17 12:57:18,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:18,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,941 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 90938baa8b6f4fe1b0e62338e4f40807, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681736236653 2023-04-17 12:57:18,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:18,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,942 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5d0ac188e494417784e29bcdf6671707, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,945 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736238934, max timestamp=1681736238945Num uniques 859; 2023-04-17 12:57:18,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:57:18,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:18,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:18,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:18,949 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:18,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:18,949 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,950 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:18,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736238934, max timestamp=1681736238945Num uniques 859; 2023-04-17 12:57:18,950 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:18,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:57:18,950 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:18,951 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:18,951 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:18,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736238934, max timestamp=1681736238945Num uniques 859; 2023-04-17 12:57:18,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:57:18,950 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:18,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:18,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:18,966 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:18,991 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/41ab8e4d866a4c6cb7607dc2b775ffd6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/41ab8e4d866a4c6cb7607dc2b775ffd6 2023-04-17 12:57:18,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736298991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:18,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:18,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736298996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736298998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,000 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 41ab8e4d866a4c6cb7607dc2b775ffd6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:19,000 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:19,001 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736238792; duration=0sec 2023-04-17 12:57:19,001 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:19,052 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:19,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:19,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:19,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736299100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736299140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736299140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:19,151 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:19,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:19,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:19,152 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,152 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,219 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:19,219 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,220 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,220 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:19,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:19,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:19,222 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:19,222 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,222 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:19,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:19,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:19,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:19,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:19,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:19,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736299242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,297 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:19,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee04349[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:19,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:19,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,302 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,302 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:19,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:19,302 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:19,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:19,304 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:19,305 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:19,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:19,307 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:19,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:19,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:19,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:19,307 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:19,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736299303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:19,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,322 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/3956fc2eddff4b3ca730a96ba93e1b7d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/3956fc2eddff4b3ca730a96ba93e1b7d 2023-04-17 12:57:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736299343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736299343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/223d0053789c480aa8c27f3f33aa968f 2023-04-17 12:57:19,489 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 3956fc2eddff4b3ca730a96ba93e1b7d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:19,489 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:19,490 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736238791; duration=0sec 2023-04-17 12:57:19,490 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:19,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:19,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:19,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:19,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:19,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,492 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:19,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:19,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:19,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,492 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:19,492 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:19,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:19,592 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:19,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:19,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:19,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736299612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:19,622 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:19,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:19,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:19,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:19,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:19,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:19,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:19,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:19,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:19,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:19,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/2fd4547b44c34ec69aea2c38601a6588 2023-04-17 12:57:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736299646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:19,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736299648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,662 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:19,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:19,663 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:19,663 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:19,663 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:19,663 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:19,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:19,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:19,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:19,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:19,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,673 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e856c08[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:19,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:19,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,677 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:19,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:19,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:19,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:19,679 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:19,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,679 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,679 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:19,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:19,682 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:19,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:19,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:19,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:19,682 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:19,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:19,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:19,684 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:19,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:19,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:19,688 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:19,688 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:19,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:19,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:19,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/3b5161917bf74da593c47cffaaf4f8ac 2023-04-17 12:57:19,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/223d0053789c480aa8c27f3f33aa968f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f 2023-04-17 12:57:19,734 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f, entries=150, sequenceid=495, filesize=11.9 K 2023-04-17 12:57:19,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/2fd4547b44c34ec69aea2c38601a6588 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588 2023-04-17 12:57:19,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588, entries=150, sequenceid=495, filesize=11.9 K 2023-04-17 12:57:19,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/3b5161917bf74da593c47cffaaf4f8ac as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac 2023-04-17 12:57:19,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac, entries=150, sequenceid=495, filesize=11.9 K 2023-04-17 12:57:19,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for f14a4c6d8659d8cf4158b082301a3bb3 in 873ms, sequenceid=495, compaction requested=false 2023-04-17 12:57:19,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:19,788 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:19,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:19,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:19,796 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:19,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:19,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:19,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:19,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:19,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:19,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:19,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:19,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:19,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:19,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:19,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:19,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:19,816 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:19,816 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:19,816 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:19,816 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:19,816 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-17 12:57:19,821 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:19,822 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:19,822 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:19,822 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:19,822 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:19,822 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:19,901 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/86043b0ffedb4008b68086012a42f4b0 2023-04-17 12:57:19,965 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/2a77970f2c08478e8fbcf7902c10dee0 2023-04-17 12:57:20,033 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/79a7dc967042478aaaafdc437076ce2a 2023-04-17 12:57:20,040 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/86043b0ffedb4008b68086012a42f4b0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0 2023-04-17 12:57:20,050 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0, entries=150, sequenceid=531, filesize=11.9 K 2023-04-17 12:57:20,052 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/2a77970f2c08478e8fbcf7902c10dee0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0 2023-04-17 12:57:20,061 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0, entries=150, sequenceid=531, filesize=11.9 K 2023-04-17 12:57:20,062 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/79a7dc967042478aaaafdc437076ce2a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a 2023-04-17 12:57:20,070 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a, entries=150, sequenceid=531, filesize=11.9 K 2023-04-17 12:57:20,071 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=0 B/0 for f14a4c6d8659d8cf4158b082301a3bb3 in 255ms, sequenceid=531, compaction requested=true 2023-04-17 12:57:20,071 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:20,071 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,071 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:20,071 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:20,071 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,071 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:20,071 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:20,073 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:20,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6da440[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:20,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:20,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:20,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:20,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:20,078 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,078 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:20,078 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,078 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:20,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:20,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:20,082 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:20,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:20,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:20,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:20,082 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:20,085 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,087 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,087 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,096 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:20,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:20,097 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:20,097 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:20,180 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:20,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:20,197 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:20,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/756f4a5b73b04ca58ee8da4dfb7cafc0 2023-04-17 12:57:20,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:20,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:20,215 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:20,215 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,215 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:20,215 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:20,215 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,313 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:20,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:20,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:20,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:20,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:20,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:20,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:20,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:20,439 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:20,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:20,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:20,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:20,441 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,441 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:20,441 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:20,439 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 122 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:20,442 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 124 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:20,446 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,448 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,448 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:20,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72792ec5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:20,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:20,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,569 INFO [AsyncFSWAL-0-hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16-prefix:jenkins-hbase17.apache.org,40215,1681736213280] wal.AbstractFSWAL(1141): Slow sync cost: 118 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:33379,DS-09032929-f765-4ade-a376-e34520985675,DISK]] 2023-04-17 12:57:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736300443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736240309, max timestamp=1681736240450Num uniques 859; 2023-04-17 12:57:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736300445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:20,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736240309, max timestamp=1681736240450Num uniques 859; 2023-04-17 12:57:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736240309, max timestamp=1681736240450Num uniques 859; 2023-04-17 12:57:20,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:20,588 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:20,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:20,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:20,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:20,597 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:20,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:20,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:20,613 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:20,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:20,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:20,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:20,613 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:20,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,622 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,622 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:20,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:20,639 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:20,639 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:20,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,639 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,639 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:20,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736300640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736300671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736300672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/07a96fc5bd23491797de2ca3c652d90b 2023-04-17 12:57:20,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/b1ae6c7ecb0c40eb8b7e542c753245dc 2023-04-17 12:57:20,739 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:20,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:20,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:20,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:20,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:20,744 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:20,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:20,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:20,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,745 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:20,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:20,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:20,747 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:20,747 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:20,748 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/756f4a5b73b04ca58ee8da4dfb7cafc0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0 2023-04-17 12:57:20,748 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,748 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:20,748 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:20,748 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:20,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:20,749 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:20,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e6fd1c8[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:20,753 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:20,753 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,754 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:20,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:20,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:20,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:20,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736300754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0, entries=200, sequenceid=542, filesize=14.3 K 2023-04-17 12:57:20,756 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:20,757 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,757 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/07a96fc5bd23491797de2ca3c652d90b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b 2023-04-17 12:57:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:20,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:20,769 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:20,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:20,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:20,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:20,769 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:20,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b, entries=150, sequenceid=542, filesize=11.9 K 2023-04-17 12:57:20,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/b1ae6c7ecb0c40eb8b7e542c753245dc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc 2023-04-17 12:57:20,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:20,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,780 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,780 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,780 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:20,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,781 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:20,781 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc, entries=150, sequenceid=542, filesize=11.9 K 2023-04-17 12:57:20,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f14a4c6d8659d8cf4158b082301a3bb3 in 608ms, sequenceid=542, compaction requested=true 2023-04-17 12:57:20,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:20,788 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:20,788 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:20,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:20,790 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:20,790 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:20,790 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,790 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=48.4 K 2023-04-17 12:57:20,790 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:20,791 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:20,791 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,791 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/3956fc2eddff4b3ca730a96ba93e1b7d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=50.8 K 2023-04-17 12:57:20,791 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b1ad62d7a1ad4a9eb1f8c3b461c043fe, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:20,791 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 2fd4547b44c34ec69aea2c38601a6588, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681736238223 2023-04-17 12:57:20,792 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3956fc2eddff4b3ca730a96ba93e1b7d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:20,792 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 2a77970f2c08478e8fbcf7902c10dee0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681736238983 2023-04-17 12:57:20,792 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 223d0053789c480aa8c27f3f33aa968f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681736238223 2023-04-17 12:57:20,792 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 07a96fc5bd23491797de2ca3c652d90b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:20,792 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 86043b0ffedb4008b68086012a42f4b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681736238983 2023-04-17 12:57:20,793 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 756f4a5b73b04ca58ee8da4dfb7cafc0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:20,819 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#87 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:20,823 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#88 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:20,843 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/b75a34de03c54cffb9c63651ca00416e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b75a34de03c54cffb9c63651ca00416e 2023-04-17 12:57:20,855 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into b75a34de03c54cffb9c63651ca00416e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:20,855 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:20,855 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=12, startTime=1681736240788; duration=0sec 2023-04-17 12:57:20,855 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:20,855 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:20,856 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:20,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:20,857 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,857 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/41ab8e4d866a4c6cb7607dc2b775ffd6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=48.4 K 2023-04-17 12:57:20,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 41ab8e4d866a4c6cb7607dc2b775ffd6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1681736237084 2023-04-17 12:57:20,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3b5161917bf74da593c47cffaaf4f8ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681736238223 2023-04-17 12:57:20,858 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 79a7dc967042478aaaafdc437076ce2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1681736238983 2023-04-17 12:57:20,858 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b1ae6c7ecb0c40eb8b7e542c753245dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:20,865 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:20,875 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/bf7812926cd94e96bf54044f9f552064 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/bf7812926cd94e96bf54044f9f552064 2023-04-17 12:57:20,881 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:20,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:20,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:20,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:20,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:20,886 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into bf7812926cd94e96bf54044f9f552064(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:20,886 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:20,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:20,886 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=12, startTime=1681736240788; duration=0sec 2023-04-17 12:57:20,886 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:20,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:20,898 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:20,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:20,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:20,902 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,902 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,903 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,903 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,904 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,904 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:20,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:20,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:20,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:20,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:20,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:20,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:20,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:20,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:20,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:20,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:20,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:20,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:20,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:20,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/8ecfe71d13f64c24ae78035e8f314683 2023-04-17 12:57:20,917 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736300917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:20,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7f0b21[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:20,919 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:20,919 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,920 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:20,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:20,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:20,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736300918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:20,922 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:20,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:20,925 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:20,925 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:20,925 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:20,925 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:20,925 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:20,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:20,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,929 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,931 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,936 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:20,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8648c19f6d644e009869f1838cf04cdf 2023-04-17 12:57:20,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:20,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:20,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:20,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:20,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:20,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:20,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:20,938 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:20,938 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:20,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:20,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736300958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736301020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736301022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,038 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:21,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:21,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:21,041 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:21,041 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,041 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:21,041 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:21,042 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,042 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,043 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,043 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,043 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:21,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:21,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:21,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:21,045 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:21,045 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,045 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:21,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:21,047 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,047 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:21,048 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:21,048 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:21,048 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:21,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:21,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:21,048 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:21,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:21,054 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:21,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c17edaa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:21,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:21,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,060 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,060 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:21,060 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:21,060 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:21,060 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,060 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:21,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:21,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,061 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:21,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:21,069 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:21,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:21,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:21,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:21,069 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:21,071 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,071 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,071 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:21,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:21,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,078 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:21,078 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:21,077 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,078 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,078 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,078 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,078 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,178 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:21,181 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:21,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:21,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:21,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,182 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,183 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,183 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,184 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:21,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:21,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:21,186 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:21,186 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,186 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:21,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:21,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:21,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:21,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:21,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:21,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:21,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:21,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:21,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:21,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a3edeb[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:21,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:21,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:21,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:21,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:21,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:21,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:21,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:21,199 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:21,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:21,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:21,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:21,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:21,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:21,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:21,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,206 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,206 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:21,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736301224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736301226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736301260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33660 deadline: 1681736301271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,273 DEBUG [Thread-142] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:21,290 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/fa3533bda69b4f5db3d74b40dda18011 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/fa3533bda69b4f5db3d74b40dda18011 2023-04-17 12:57:21,295 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into fa3533bda69b4f5db3d74b40dda18011(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:21,295 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:21,295 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=12, startTime=1681736240788; duration=0sec 2023-04-17 12:57:21,295 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:21,306 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:21,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:21,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:21,310 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:21,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:21,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:21,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,312 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,312 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:21,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:21,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:21,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:21,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:21,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:21,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:21,313 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,313 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:21,314 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:21,314 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:21,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:21,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:21,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:21,315 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:21,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17a28346[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:21,317 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:21,317 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,317 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:21,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:21,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:21,318 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,318 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:21,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,319 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,319 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,319 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:21,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:21,321 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:21,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:21,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:21,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:21,321 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:21,322 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,322 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,322 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:21,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,328 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:21,328 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/439a3a350d9a41f6800aecd4a4a6bb26 2023-04-17 12:57:21,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/8ecfe71d13f64c24ae78035e8f314683 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683 2023-04-17 12:57:21,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683, entries=200, sequenceid=580, filesize=14.3 K 2023-04-17 12:57:21,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/8648c19f6d644e009869f1838cf04cdf as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf 2023-04-17 12:57:21,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf, entries=150, sequenceid=580, filesize=11.9 K 2023-04-17 12:57:21,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/439a3a350d9a41f6800aecd4a4a6bb26 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26 2023-04-17 12:57:21,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26, entries=150, sequenceid=580, filesize=11.9 K 2023-04-17 12:57:21,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for f14a4c6d8659d8cf4158b082301a3bb3 in 501ms, sequenceid=580, compaction requested=false 2023-04-17 12:57:21,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:21,428 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:21,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:57:21,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:21,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:21,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:21,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:21,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:21,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:21,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:21,432 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:21,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:21,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:21,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,434 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,434 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,435 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,435 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:21,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:21,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:21,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:21,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:21,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:21,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:21,437 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,437 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:21,437 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:21,437 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:21,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:21,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:21,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:21,441 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,441 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,443 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,443 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:21,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:21,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417a490[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:21,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:21,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:21,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:21,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:21,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/2e71ee807b504346b424f8e64845134a 2023-04-17 12:57:21,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:21,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:21,457 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:21,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:21,462 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:21,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:21,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:21,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:21,463 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:21,464 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,464 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,464 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,465 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,472 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:21,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:21,472 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:21,472 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:21,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:21,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736301484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/e8280cd64b4548d7a5362671184f2e1c 2023-04-17 12:57:21,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736301526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736301532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/d6103b2fe0cc4d078810e281dfd65399 2023-04-17 12:57:21,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/2e71ee807b504346b424f8e64845134a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a 2023-04-17 12:57:21,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a, entries=150, sequenceid=594, filesize=11.9 K 2023-04-17 12:57:21,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/e8280cd64b4548d7a5362671184f2e1c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c 2023-04-17 12:57:21,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c, entries=100, sequenceid=594, filesize=9.5 K 2023-04-17 12:57:21,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/d6103b2fe0cc4d078810e281dfd65399 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399 2023-04-17 12:57:21,573 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399, entries=100, sequenceid=594, filesize=9.5 K 2023-04-17 12:57:21,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f14a4c6d8659d8cf4158b082301a3bb3 in 146ms, sequenceid=594, compaction requested=true 2023-04-17 12:57:21,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:21,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:21,575 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:21,575 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:21,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:21,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39982 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:21,577 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,577 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,577 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b75a34de03c54cffb9c63651ca00416e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=39.0 K 2023-04-17 12:57:21,577 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/bf7812926cd94e96bf54044f9f552064, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.3 K 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b75a34de03c54cffb9c63651ca00416e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:21,577 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting bf7812926cd94e96bf54044f9f552064, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:21,578 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8ecfe71d13f64c24ae78035e8f314683, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681736240617 2023-04-17 12:57:21,578 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8648c19f6d644e009869f1838cf04cdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681736240633 2023-04-17 12:57:21,578 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2e71ee807b504346b424f8e64845134a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240915 2023-04-17 12:57:21,578 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e8280cd64b4548d7a5362671184f2e1c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240915 2023-04-17 12:57:21,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:21,591 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:21,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:21,598 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:21,598 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,598 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:21,598 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:21,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:21,609 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:21,609 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:21,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:21,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:21,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:21,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:21,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:21,622 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:21,622 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:21,622 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,622 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. started... 2023-04-17 12:57:21,622 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. as already flushing 2023-04-17 12:57:21,622 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:21,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:21,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736301623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:21,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:21,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@779e7305[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:21,637 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:21,637 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:21,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:21,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:21,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:21,639 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,639 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:21,639 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,639 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:21,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:21,641 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:21,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:21,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:21,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:21,641 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:21,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:21,643 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,645 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:21,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:21,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:21,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:21,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:21,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:21,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:21,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:21,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:21,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:21,647 DEBUG [Thread-152] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:21,647 DEBUG [Thread-152] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:21,679 DEBUG [Thread-153] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29be3c5c to 127.0.0.1:62647 2023-04-17 12:57:21,679 DEBUG [Thread-153] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:21,679 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/0a58ebca78234922ac0bbecf45ab42d7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0a58ebca78234922ac0bbecf45ab42d7 2023-04-17 12:57:21,683 DEBUG [Thread-157] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fed686f to 127.0.0.1:62647 2023-04-17 12:57:21,683 DEBUG [Thread-157] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:21,685 DEBUG [Thread-155] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbd52be to 127.0.0.1:62647 2023-04-17 12:57:21,685 DEBUG [Thread-155] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:21,687 DEBUG [Thread-159] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e044e22 to 127.0.0.1:62647 2023-04-17 12:57:21,688 DEBUG [Thread-159] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:21,694 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into 0a58ebca78234922ac0bbecf45ab42d7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:21,694 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:21,694 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736241575; duration=0sec 2023-04-17 12:57:21,694 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:21,694 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:21,695 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:21,695 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:21,695 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:21,695 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/fa3533bda69b4f5db3d74b40dda18011, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=34.3 K 2023-04-17 12:57:21,696 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting fa3533bda69b4f5db3d74b40dda18011, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1681736240177 2023-04-17 12:57:21,696 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 439a3a350d9a41f6800aecd4a4a6bb26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681736240633 2023-04-17 12:57:21,697 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d6103b2fe0cc4d078810e281dfd65399, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240915 2023-04-17 12:57:21,703 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:21,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736301729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,747 DEBUG [Thread-152] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:21,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33624 deadline: 1681736301765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:21,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:21,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736301931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:22,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:22,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33662 deadline: 1681736302028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:22,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33666 deadline: 1681736302040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:22,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/64a4bf6930be40008667042bbe8e55f8 2023-04-17 12:57:22,092 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/b9a6ebb1cef1475c8545f34eb24dd0c9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b9a6ebb1cef1475c8545f34eb24dd0c9 2023-04-17 12:57:22,099 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into b9a6ebb1cef1475c8545f34eb24dd0c9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:22,100 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:22,100 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736241575; duration=0sec 2023-04-17 12:57:22,100 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:22,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/b0852afa815a49469e92f335b1ced499 2023-04-17 12:57:22,119 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/9ea658fb984d435f9a60c77ccd571861 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9ea658fb984d435f9a60c77ccd571861 2023-04-17 12:57:22,127 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into 9ea658fb984d435f9a60c77ccd571861(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:22,127 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:22,127 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736241576; duration=0sec 2023-04-17 12:57:22,127 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:22,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:22,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33678 deadline: 1681736302232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:22,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/800902cf6f2d43eba8be85628a83798e 2023-04-17 12:57:22,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/64a4bf6930be40008667042bbe8e55f8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8 2023-04-17 12:57:22,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8, entries=200, sequenceid=615, filesize=14.3 K 2023-04-17 12:57:22,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/b0852afa815a49469e92f335b1ced499 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499 2023-04-17 12:57:22,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499, entries=150, sequenceid=615, filesize=11.9 K 2023-04-17 12:57:22,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/800902cf6f2d43eba8be85628a83798e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e 2023-04-17 12:57:22,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e, entries=150, sequenceid=615, filesize=11.9 K 2023-04-17 12:57:22,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for f14a4c6d8659d8cf4158b082301a3bb3 in 968ms, sequenceid=615, compaction requested=false 2023-04-17 12:57:22,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:22,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:57:22,737 DEBUG [Thread-148] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45864842 to 127.0.0.1:62647 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:22,737 DEBUG [Thread-148] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:22,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:22,768 DEBUG [Thread-150] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4654132d to 127.0.0.1:62647 2023-04-17 12:57:22,768 DEBUG [Thread-150] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:23,035 DEBUG [Thread-146] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77443fb0 to 127.0.0.1:62647 2023-04-17 12:57:23,035 DEBUG [Thread-146] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:23,046 DEBUG [Thread-144] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x449cbd3d to 127.0.0.1:62647 2023-04-17 12:57:23,046 DEBUG [Thread-144] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:23,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/97d6ebc32857414981616d44a1cb59b5 2023-04-17 12:57:23,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/4f9c1825ef7d45c9bcc789349ead3866 2023-04-17 12:57:23,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f807c84638ac4eeabadc8c9f7e9c3228 2023-04-17 12:57:23,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/97d6ebc32857414981616d44a1cb59b5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5 2023-04-17 12:57:23,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5, entries=150, sequenceid=630, filesize=11.9 K 2023-04-17 12:57:23,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/4f9c1825ef7d45c9bcc789349ead3866 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866 2023-04-17 12:57:24,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866, entries=150, sequenceid=630, filesize=11.9 K 2023-04-17 12:57:24,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f807c84638ac4eeabadc8c9f7e9c3228 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228 2023-04-17 12:57:24,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228, entries=150, sequenceid=630, filesize=11.9 K 2023-04-17 12:57:24,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for f14a4c6d8659d8cf4158b082301a3bb3 in 1274ms, sequenceid=630, compaction requested=true 2023-04-17 12:57:24,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:24,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:24,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:24,011 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:24,011 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:24,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:24,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:24,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/A is initiating minor compaction (all files) 2023-04-17 12:57:24,012 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:24,012 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/A in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:24,012 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/B is initiating minor compaction (all files) 2023-04-17 12:57:24,012 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b9a6ebb1cef1475c8545f34eb24dd0c9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=39.1 K 2023-04-17 12:57:24,012 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/B in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:24,012 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0a58ebca78234922ac0bbecf45ab42d7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.8 K 2023-04-17 12:57:24,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b9a6ebb1cef1475c8545f34eb24dd0c9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240880 2023-04-17 12:57:24,013 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 0a58ebca78234922ac0bbecf45ab42d7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240880 2023-04-17 12:57:24,013 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 64a4bf6930be40008667042bbe8e55f8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681736241476 2023-04-17 12:57:24,013 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b0852afa815a49469e92f335b1ced499, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681736241476 2023-04-17 12:57:24,013 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 97d6ebc32857414981616d44a1cb59b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681736241616 2023-04-17 12:57:24,013 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4f9c1825ef7d45c9bcc789349ead3866, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681736241616 2023-04-17 12:57:24,035 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#B#compaction#105 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:24,036 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#A#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:24,451 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/e089d13aaec34d7395641cc93e723469 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e089d13aaec34d7395641cc93e723469 2023-04-17 12:57:24,451 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/56056accb65d45c48b692606f5941bf4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/56056accb65d45c48b692606f5941bf4 2023-04-17 12:57:24,456 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/B of f14a4c6d8659d8cf4158b082301a3bb3 into e089d13aaec34d7395641cc93e723469(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:24,456 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/A of f14a4c6d8659d8cf4158b082301a3bb3 into 56056accb65d45c48b692606f5941bf4(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:24,456 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:24,456 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:24,456 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/B, priority=13, startTime=1681736244011; duration=0sec 2023-04-17 12:57:24,456 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/A, priority=13, startTime=1681736244011; duration=0sec 2023-04-17 12:57:24,456 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:24,456 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:24,456 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:24,457 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:24,457 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): f14a4c6d8659d8cf4158b082301a3bb3/C is initiating minor compaction (all files) 2023-04-17 12:57:24,457 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f14a4c6d8659d8cf4158b082301a3bb3/C in TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:24,457 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9ea658fb984d435f9a60c77ccd571861, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp, totalSize=36.8 K 2023-04-17 12:57:24,457 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9ea658fb984d435f9a60c77ccd571861, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681736240880 2023-04-17 12:57:24,458 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 800902cf6f2d43eba8be85628a83798e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681736241476 2023-04-17 12:57:24,458 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f807c84638ac4eeabadc8c9f7e9c3228, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681736241616 2023-04-17 12:57:24,464 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): f14a4c6d8659d8cf4158b082301a3bb3#C#compaction#107 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:24,878 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/aa80779c2e2a4f2b893f9f755c760145 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aa80779c2e2a4f2b893f9f755c760145 2023-04-17 12:57:24,886 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f14a4c6d8659d8cf4158b082301a3bb3/C of f14a4c6d8659d8cf4158b082301a3bb3 into aa80779c2e2a4f2b893f9f755c760145(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:24,887 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:24,887 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3., storeName=f14a4c6d8659d8cf4158b082301a3bb3/C, priority=13, startTime=1681736244011; duration=0sec 2023-04-17 12:57:24,887 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:25,315 DEBUG [Thread-142] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x202b48c4 to 127.0.0.1:62647 2023-04-17 12:57:25,315 DEBUG [Thread-142] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:25,315 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:57:25,315 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-17 12:57:25,315 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-17 12:57:25,315 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 3548 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 3552 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1603 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 4808 rows 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1505 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 4514 rows 2023-04-17 12:57:25,316 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:57:25,316 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x169fe8bf to 127.0.0.1:62647 2023-04-17 12:57:25,316 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:25,321 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:57:25,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:57:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:25,336 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736245336"}]},"ts":"1681736245336"} 2023-04-17 12:57:25,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:25,338 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:57:25,419 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:57:25,421 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, UNASSIGN}] 2023-04-17 12:57:25,423 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, UNASSIGN 2023-04-17 12:57:25,424 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f14a4c6d8659d8cf4158b082301a3bb3, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:25,424 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736245424"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736245424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736245424"}]},"ts":"1681736245424"} 2023-04-17 12:57:25,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:25,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:25,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:25,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing f14a4c6d8659d8cf4158b082301a3bb3, disabling compactions & flushes 2023-04-17 12:57:25,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:25,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:25,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. after waiting 0 ms 2023-04-17 12:57:25,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:25,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing f14a4c6d8659d8cf4158b082301a3bb3 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=A 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=B 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f14a4c6d8659d8cf4158b082301a3bb3, store=C 2023-04-17 12:57:25,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:25,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:25,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:25,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/7b91077ad7684a2ba786a69578dce6d0 2023-04-17 12:57:26,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/fa08c43c817e44079d10fc8a0c4647e3 2023-04-17 12:57:26,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f364e353f5b9468f817d4c677c9d4db7 2023-04-17 12:57:26,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:26,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/A/7b91077ad7684a2ba786a69578dce6d0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7b91077ad7684a2ba786a69578dce6d0 2023-04-17 12:57:26,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7b91077ad7684a2ba786a69578dce6d0, entries=150, sequenceid=640, filesize=11.9 K 2023-04-17 12:57:26,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/B/fa08c43c817e44079d10fc8a0c4647e3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/fa08c43c817e44079d10fc8a0c4647e3 2023-04-17 12:57:26,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/fa08c43c817e44079d10fc8a0c4647e3, entries=150, sequenceid=640, filesize=11.9 K 2023-04-17 12:57:26,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/.tmp/C/f364e353f5b9468f817d4c677c9d4db7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f364e353f5b9468f817d4c677c9d4db7 2023-04-17 12:57:26,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f364e353f5b9468f817d4c677c9d4db7, entries=150, sequenceid=640, filesize=11.9 K 2023-04-17 12:57:26,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for f14a4c6d8659d8cf4158b082301a3bb3 in 885ms, sequenceid=640, compaction requested=false 2023-04-17 12:57:26,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/23af23720fe04fa88581ab99494bf338, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/a642c39bfc1544d7ac24dacf0bdae8c8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/34b580a9c36942859844329ef2b5a941, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/71332b7be7224521ba74491b3b443498, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/be33ade6dfc14366bd534e2b38a40229, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8d6ebc05da35423b92dc22986eaae53a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/831d4ed6a7224b26ba8d2f6a539e8bbf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/3956fc2eddff4b3ca730a96ba93e1b7d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b75a34de03c54cffb9c63651ca00416e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b9a6ebb1cef1475c8545f34eb24dd0c9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5] to archive 2023-04-17 12:57:26,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:26,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/5f01686d8acd4fc2aa4c5cfc0221f918 2023-04-17 12:57:26,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/256138f538bc4e138c316a4558ea8184 2023-04-17 12:57:26,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/88ed0746516d4bf089658cbe32d84a40 2023-04-17 12:57:26,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/23af23720fe04fa88581ab99494bf338 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/23af23720fe04fa88581ab99494bf338 2023-04-17 12:57:26,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ec259a3a7b31461196475ec916d89048 2023-04-17 12:57:26,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/a642c39bfc1544d7ac24dacf0bdae8c8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/a642c39bfc1544d7ac24dacf0bdae8c8 2023-04-17 12:57:26,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/f57054ae48ec4d8897c47294c8cfda16 2023-04-17 12:57:26,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/de6c4bf765c54da88243080ca5d9a8a8 2023-04-17 12:57:26,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/34b580a9c36942859844329ef2b5a941 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/34b580a9c36942859844329ef2b5a941 2023-04-17 12:57:26,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/0f5e0f6e208d4b918f50ac51dfdb045a 2023-04-17 12:57:26,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/ab30b01260a84faf9ab515e631af38be 2023-04-17 12:57:26,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/e369d1c590ec424a86e49776c4f85adb 2023-04-17 12:57:26,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/71332b7be7224521ba74491b3b443498 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/71332b7be7224521ba74491b3b443498 2023-04-17 12:57:26,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8dd415abd5ba49889c062c4ec2df674d 2023-04-17 12:57:26,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/be33ade6dfc14366bd534e2b38a40229 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/be33ade6dfc14366bd534e2b38a40229 2023-04-17 12:57:26,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/d1b21f1afa0c487ab5a2653a47add366 2023-04-17 12:57:26,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/cddcc5780eac4cde9ac290d05989c9de 2023-04-17 12:57:26,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7e7d657caf97449bab82693226a398c8 2023-04-17 12:57:26,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/9ced730983eb49a5b83d09d131ea9f63 2023-04-17 12:57:26,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8d6ebc05da35423b92dc22986eaae53a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8d6ebc05da35423b92dc22986eaae53a 2023-04-17 12:57:26,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/24559f6ce54c4423a32a115a728e8317 2023-04-17 12:57:26,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/69a8825a771f44faa4c0e7591d3cdac9 2023-04-17 12:57:26,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/831d4ed6a7224b26ba8d2f6a539e8bbf to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/831d4ed6a7224b26ba8d2f6a539e8bbf 2023-04-17 12:57:26,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2ebd227094ce4183a414bef73a5b4646 2023-04-17 12:57:26,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/3956fc2eddff4b3ca730a96ba93e1b7d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/3956fc2eddff4b3ca730a96ba93e1b7d 2023-04-17 12:57:26,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/98118140dec642069cf7af47f7648f8c 2023-04-17 12:57:26,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/223d0053789c480aa8c27f3f33aa968f 2023-04-17 12:57:26,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/86043b0ffedb4008b68086012a42f4b0 2023-04-17 12:57:26,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/756f4a5b73b04ca58ee8da4dfb7cafc0 2023-04-17 12:57:26,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b75a34de03c54cffb9c63651ca00416e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b75a34de03c54cffb9c63651ca00416e 2023-04-17 12:57:26,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/8ecfe71d13f64c24ae78035e8f314683 2023-04-17 12:57:26,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b9a6ebb1cef1475c8545f34eb24dd0c9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/b9a6ebb1cef1475c8545f34eb24dd0c9 2023-04-17 12:57:26,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/2e71ee807b504346b424f8e64845134a 2023-04-17 12:57:26,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/64a4bf6930be40008667042bbe8e55f8 2023-04-17 12:57:26,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/97d6ebc32857414981616d44a1cb59b5 2023-04-17 12:57:26,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4b1c7cad892f4b859cd4a0df9d43c13c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/66c5216bd781477198d9502ebb8a9721, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/cf1d2d88b24d4433a85ec0f9fe7790c9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eb355024f154aea8b39341e29c9808a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/475d95c20d7c4f4d844aef6e75f1b428, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/5b06c8b9ff924194b5bf716027afec25, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/7d8e0c43ae024c88b10c1e5423005e03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/bf7812926cd94e96bf54044f9f552064, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0a58ebca78234922ac0bbecf45ab42d7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866] to archive 2023-04-17 12:57:26,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:26,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/05cf3992a4e9499c969d8b92876a111c 2023-04-17 12:57:26,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c27db47f875b4c4e88d0c57f1ef7c598 2023-04-17 12:57:26,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4b1c7cad892f4b859cd4a0df9d43c13c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4b1c7cad892f4b859cd4a0df9d43c13c 2023-04-17 12:57:26,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/a2e8aa8fea5a457d8fe8aeab4b3bb261 2023-04-17 12:57:26,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0493d5a1c9fe4060b16de4a9fa966475 2023-04-17 12:57:26,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/66c5216bd781477198d9502ebb8a9721 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/66c5216bd781477198d9502ebb8a9721 2023-04-17 12:57:26,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/028a0a8284e8444a8d1de49e8fc80681 2023-04-17 12:57:26,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/90bea4f7f03d4b2e8aea8a9165700e81 2023-04-17 12:57:26,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/cf1d2d88b24d4433a85ec0f9fe7790c9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/cf1d2d88b24d4433a85ec0f9fe7790c9 2023-04-17 12:57:26,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/c3963d6b3bed4b15b5202f89559af3b2 2023-04-17 12:57:26,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8446adde36204f3ebe1286876d13a845 2023-04-17 12:57:26,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eb355024f154aea8b39341e29c9808a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eb355024f154aea8b39341e29c9808a 2023-04-17 12:57:26,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/6eeefa963214457aa9fc0d5b807887ae 2023-04-17 12:57:26,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/530a5949f86345f19d80a33b2c0ee4d0 2023-04-17 12:57:26,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/475d95c20d7c4f4d844aef6e75f1b428 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/475d95c20d7c4f4d844aef6e75f1b428 2023-04-17 12:57:26,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/3164f4ed71d74075a2578e3c43465d51 2023-04-17 12:57:26,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9711dd513d6840a389519fcf8d2cd7f9 2023-04-17 12:57:26,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/96ecb945c83244a28251de048b9dd290 2023-04-17 12:57:26,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/5b06c8b9ff924194b5bf716027afec25 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/5b06c8b9ff924194b5bf716027afec25 2023-04-17 12:57:26,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/ae0e72a883c545cdbeb835b15a154fe7 2023-04-17 12:57:26,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/53ed4036154f4356b55a4389ade6f493 2023-04-17 12:57:26,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/7d8e0c43ae024c88b10c1e5423005e03 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/7d8e0c43ae024c88b10c1e5423005e03 2023-04-17 12:57:26,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8651d0b9dd104bac9bd6b8a16ea111df 2023-04-17 12:57:26,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/aa73b4e0a78945dea819a3d181220db4 2023-04-17 12:57:26,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b1ad62d7a1ad4a9eb1f8c3b461c043fe 2023-04-17 12:57:26,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/9c37c280e5b24d0d96698d128beb1767 2023-04-17 12:57:26,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2fd4547b44c34ec69aea2c38601a6588 2023-04-17 12:57:26,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/2a77970f2c08478e8fbcf7902c10dee0 2023-04-17 12:57:26,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/bf7812926cd94e96bf54044f9f552064 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/bf7812926cd94e96bf54044f9f552064 2023-04-17 12:57:26,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/07a96fc5bd23491797de2ca3c652d90b 2023-04-17 12:57:26,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/8648c19f6d644e009869f1838cf04cdf 2023-04-17 12:57:26,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0a58ebca78234922ac0bbecf45ab42d7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/0a58ebca78234922ac0bbecf45ab42d7 2023-04-17 12:57:26,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e8280cd64b4548d7a5362671184f2e1c 2023-04-17 12:57:26,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/b0852afa815a49469e92f335b1ced499 2023-04-17 12:57:26,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/4f9c1825ef7d45c9bcc789349ead3866 2023-04-17 12:57:26,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c48d70fe72f7490983b890600d9606d2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/985a84565cef481980e4c02861c7e376, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/135c0ae86ecd4ec7ad5e291780fc1b5c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/36ed33d3bb634778ba6997c0bd11718a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/eaf6dd6359a74dc5a540ce3b9c61c154, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/97bfefb739514e7091a017c20863e59e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/992b7ed41f1045a89d30a0e6535b947e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/41ab8e4d866a4c6cb7607dc2b775ffd6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/fa3533bda69b4f5db3d74b40dda18011, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9ea658fb984d435f9a60c77ccd571861, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228] to archive 2023-04-17 12:57:26,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:26,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9f8102c5f8cf409c9ace2b01498aa986 2023-04-17 12:57:26,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6b784f02850d44d292ac9e7816ef1229 2023-04-17 12:57:26,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c48d70fe72f7490983b890600d9606d2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c48d70fe72f7490983b890600d9606d2 2023-04-17 12:57:26,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e8e46002104b4fd083603d10588bf8e6 2023-04-17 12:57:26,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f6ac3a7f4a334a8899550481502034df 2023-04-17 12:57:26,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/985a84565cef481980e4c02861c7e376 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/985a84565cef481980e4c02861c7e376 2023-04-17 12:57:26,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/c8de98149a5b40fc9ccd3d619ae1256f 2023-04-17 12:57:26,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aea849aed3c148bdb99d4edcb61481d0 2023-04-17 12:57:26,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/135c0ae86ecd4ec7ad5e291780fc1b5c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/135c0ae86ecd4ec7ad5e291780fc1b5c 2023-04-17 12:57:26,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/694fa975113c43b68b2c64c50833e583 2023-04-17 12:57:26,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/e993c0f9217e41a0a34177eaba2217cc 2023-04-17 12:57:26,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/36ed33d3bb634778ba6997c0bd11718a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/36ed33d3bb634778ba6997c0bd11718a 2023-04-17 12:57:26,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a4a1343b4d194f9dab8c571718adf532 2023-04-17 12:57:26,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/0261be9ef9e9407aaf4394586f923eb2 2023-04-17 12:57:26,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/eaf6dd6359a74dc5a540ce3b9c61c154 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/eaf6dd6359a74dc5a540ce3b9c61c154 2023-04-17 12:57:26,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/ce06a90d2be347208d3a7e21eb74c97b 2023-04-17 12:57:26,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90f22b9f4cb44244b9e8452112862956 2023-04-17 12:57:26,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/a3fe340b8c1a4e1fb76765ae2e3fda88 2023-04-17 12:57:26,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/97bfefb739514e7091a017c20863e59e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/97bfefb739514e7091a017c20863e59e 2023-04-17 12:57:26,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b4cbffe7f12c4472abf9f1c3a6e2decf 2023-04-17 12:57:26,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5ad8e4bd58744f20997c3458aee08440 2023-04-17 12:57:26,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/992b7ed41f1045a89d30a0e6535b947e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/992b7ed41f1045a89d30a0e6535b947e 2023-04-17 12:57:26,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/6ac98a6927924d7ea713ea6f4511f46a 2023-04-17 12:57:26,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/90938baa8b6f4fe1b0e62338e4f40807 2023-04-17 12:57:26,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/41ab8e4d866a4c6cb7607dc2b775ffd6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/41ab8e4d866a4c6cb7607dc2b775ffd6 2023-04-17 12:57:26,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/5d0ac188e494417784e29bcdf6671707 2023-04-17 12:57:26,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/3b5161917bf74da593c47cffaaf4f8ac 2023-04-17 12:57:26,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/79a7dc967042478aaaafdc437076ce2a 2023-04-17 12:57:26,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/fa3533bda69b4f5db3d74b40dda18011 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/fa3533bda69b4f5db3d74b40dda18011 2023-04-17 12:57:26,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/b1ae6c7ecb0c40eb8b7e542c753245dc 2023-04-17 12:57:26,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/439a3a350d9a41f6800aecd4a4a6bb26 2023-04-17 12:57:26,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9ea658fb984d435f9a60c77ccd571861 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/9ea658fb984d435f9a60c77ccd571861 2023-04-17 12:57:26,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/d6103b2fe0cc4d078810e281dfd65399 2023-04-17 12:57:26,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/800902cf6f2d43eba8be85628a83798e 2023-04-17 12:57:26,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f807c84638ac4eeabadc8c9f7e9c3228 2023-04-17 12:57:26,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/recovered.edits/643.seqid, newMaxSeqId=643, maxSeqId=1 2023-04-17 12:57:26,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3. 2023-04-17 12:57:26,718 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for f14a4c6d8659d8cf4158b082301a3bb3: 2023-04-17 12:57:26,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:26,720 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=f14a4c6d8659d8cf4158b082301a3bb3, regionState=CLOSED 2023-04-17 12:57:26,721 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736246720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736246720"}]},"ts":"1681736246720"} 2023-04-17 12:57:26,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-17 12:57:26,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure f14a4c6d8659d8cf4158b082301a3bb3, server=jenkins-hbase17.apache.org,40215,1681736213280 in 1.2960 sec 2023-04-17 12:57:26,725 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-17 12:57:26,725 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f14a4c6d8659d8cf4158b082301a3bb3, UNASSIGN in 1.3030 sec 2023-04-17 12:57:26,726 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736246726"}]},"ts":"1681736246726"} 2023-04-17 12:57:26,727 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:57:26,728 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:57:26,730 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4020 sec 2023-04-17 12:57:27,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 12:57:27,444 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-17 12:57:27,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:57:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,454 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,455 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 12:57:27,460 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:27,463 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/recovered.edits] 2023-04-17 12:57:27,465 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/56056accb65d45c48b692606f5941bf4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/56056accb65d45c48b692606f5941bf4 2023-04-17 12:57:27,467 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7b91077ad7684a2ba786a69578dce6d0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/A/7b91077ad7684a2ba786a69578dce6d0 2023-04-17 12:57:27,469 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e089d13aaec34d7395641cc93e723469 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/e089d13aaec34d7395641cc93e723469 2023-04-17 12:57:27,470 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/fa08c43c817e44079d10fc8a0c4647e3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/B/fa08c43c817e44079d10fc8a0c4647e3 2023-04-17 12:57:27,477 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aa80779c2e2a4f2b893f9f755c760145 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/aa80779c2e2a4f2b893f9f755c760145 2023-04-17 12:57:27,478 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f364e353f5b9468f817d4c677c9d4db7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/C/f364e353f5b9468f817d4c677c9d4db7 2023-04-17 12:57:27,481 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/recovered.edits/643.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3/recovered.edits/643.seqid 2023-04-17 12:57:27,482 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/f14a4c6d8659d8cf4158b082301a3bb3 2023-04-17 12:57:27,482 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:57:27,485 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,503 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:57:27,542 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:57:27,544 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,544 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:57:27,544 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736247544"}]},"ts":"9223372036854775807"} 2023-04-17 12:57:27,547 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:57:27,547 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f14a4c6d8659d8cf4158b082301a3bb3, NAME => 'TestAcidGuarantees,,1681736220457.f14a4c6d8659d8cf4158b082301a3bb3.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:57:27,547 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:57:27,547 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736247547"}]},"ts":"9223372036854775807"} 2023-04-17 12:57:27,550 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:57:27,551 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,553 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 103 msec 2023-04-17 12:57:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 12:57:27,558 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-17 12:57:27,579 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 204) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase17:40215-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8531fa5-shared-pool-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_686803337_15 at /127.0.0.1:53676 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8531fa5-shared-pool-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8531fa5-shared-pool-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_686803337_15 at /127.0.0.1:53610 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_686803337_15 at /127.0.0.1:53630 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x8531fa5-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=583 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=573 (was 367) - SystemLoadAverage LEAK? -, ProcessCount=192 (was 194), AvailableMemoryMB=3342 (was 3842) 2023-04-17 12:57:27,591 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=573, ProcessCount=192, AvailableMemoryMB=3342 2023-04-17 12:57:27,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:57:27,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:57:27,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:27,597 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:57:27,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-17 12:57:27,599 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:57:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:27,601 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:27,601 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e empty. 2023-04-17 12:57:27,602 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:27,602 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:57:27,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:28,014 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:57:28,016 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 26c16d52fc9da7952b2d73253571802e, NAME => 'TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:57:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:28,423 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:57:28,423 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 26c16d52fc9da7952b2d73253571802e, disabling compactions & flushes 2023-04-17 12:57:28,423 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,423 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,424 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. after waiting 0 ms 2023-04-17 12:57:28,424 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,424 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,424 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:28,427 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:57:28,429 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736248428"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736248428"}]},"ts":"1681736248428"} 2023-04-17 12:57:28,430 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:57:28,431 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:57:28,432 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736248432"}]},"ts":"1681736248432"} 2023-04-17 12:57:28,433 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:57:28,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, ASSIGN}] 2023-04-17 12:57:28,438 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, ASSIGN 2023-04-17 12:57:28,439 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:57:28,590 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:28,591 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736248590"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736248590"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736248590"}]},"ts":"1681736248590"} 2023-04-17 12:57:28,594 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:28,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 26c16d52fc9da7952b2d73253571802e, NAME => 'TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:57:28,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:57:28,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,756 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,758 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A 2023-04-17 12:57:28,758 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A 2023-04-17 12:57:28,758 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:28,759 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName A 2023-04-17 12:57:28,759 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:28,760 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,761 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B 2023-04-17 12:57:28,761 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B 2023-04-17 12:57:28,761 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:28,761 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName B 2023-04-17 12:57:28,762 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:28,762 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,763 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C 2023-04-17 12:57:28,764 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C 2023-04-17 12:57:28,764 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:28,764 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName C 2023-04-17 12:57:28,764 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:28,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:57:28,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:28,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:57:28,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 26c16d52fc9da7952b2d73253571802e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61883955, jitterRate=-0.07785721123218536}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:57:28,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:28,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., pid=18, masterSystemTime=1681736248748 2023-04-17 12:57:28,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:28,776 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:28,776 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736248776"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736248776"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736248776"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736248776"}]},"ts":"1681736248776"} 2023-04-17 12:57:28,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-17 12:57:28,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 in 184 msec 2023-04-17 12:57:28,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-17 12:57:28,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, ASSIGN in 344 msec 2023-04-17 12:57:28,782 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:57:28,782 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736248782"}]},"ts":"1681736248782"} 2023-04-17 12:57:28,783 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:57:28,785 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:57:28,786 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1910 sec 2023-04-17 12:57:29,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 12:57:29,706 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-17 12:57:29,707 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7306f12a to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:29,712 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31388307, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:29,714 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:29,718 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:29,720 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:57:29,724 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:50394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:57:29,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:57:29,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:57:29,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:30,162 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:57:30,164 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:57:30,164 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:57:30,170 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 12:57:30,185 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, REOPEN/MOVE}] 2023-04-17 12:57:30,185 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, REOPEN/MOVE 2023-04-17 12:57:30,186 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:30,186 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736250186"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736250186"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736250186"}]},"ts":"1681736250186"} 2023-04-17 12:57:30,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:30,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 26c16d52fc9da7952b2d73253571802e, disabling compactions & flushes 2023-04-17 12:57:30,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. after waiting 0 ms 2023-04-17 12:57:30,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 12:57:30,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,349 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:30,349 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 26c16d52fc9da7952b2d73253571802e to self. 2023-04-17 12:57:30,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,354 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=CLOSED 2023-04-17 12:57:30,354 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736250354"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736250354"}]},"ts":"1681736250354"} 2023-04-17 12:57:30,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-17 12:57:30,359 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 in 169 msec 2023-04-17 12:57:30,360 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=true 2023-04-17 12:57:30,510 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:30,510 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736250510"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736250510"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736250510"}]},"ts":"1681736250510"} 2023-04-17 12:57:30,512 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:30,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 26c16d52fc9da7952b2d73253571802e, NAME => 'TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:57:30,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:57:30,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,677 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,684 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A 2023-04-17 12:57:30,684 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A 2023-04-17 12:57:30,685 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:30,688 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName A 2023-04-17 12:57:30,692 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:30,693 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,695 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B 2023-04-17 12:57:30,695 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B 2023-04-17 12:57:30,695 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:30,696 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName B 2023-04-17 12:57:30,697 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:30,697 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,698 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C 2023-04-17 12:57:30,698 DEBUG [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C 2023-04-17 12:57:30,699 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:57:30,700 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 26c16d52fc9da7952b2d73253571802e columnFamilyName C 2023-04-17 12:57:30,701 INFO [StoreOpener-26c16d52fc9da7952b2d73253571802e-1] regionserver.HStore(310): Store=26c16d52fc9da7952b2d73253571802e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:57:30,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:57:30,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:30,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 26c16d52fc9da7952b2d73253571802e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75128345, jitterRate=0.1194995790719986}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:57:30,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:30,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., pid=23, masterSystemTime=1681736250665 2023-04-17 12:57:30,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:30,711 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:30,712 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736250711"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736250711"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736250711"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736250711"}]},"ts":"1681736250711"} 2023-04-17 12:57:30,716 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-17 12:57:30,716 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 in 201 msec 2023-04-17 12:57:30,717 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-17 12:57:30,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, REOPEN/MOVE in 532 msec 2023-04-17 12:57:30,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-17 12:57:30,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 550 msec 2023-04-17 12:57:30,734 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 989 msec 2023-04-17 12:57:30,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-17 12:57:30,746 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5e3bbbd0 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,754 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23640038, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,755 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x2d781def to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,768 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,769 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3b3311fc to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,781 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5649f410, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,782 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7acc1387 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,791 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@633c81ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,792 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7be739cc to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,800 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61ad5d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,800 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x486fba63 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,819 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43a14bcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,820 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,833 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,833 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,842 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,843 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x52902392 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:57:30,852 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@651f745a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:57:30,901 DEBUG [hconnection-0x3c706b00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,902 DEBUG [hconnection-0x7175d325-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,904 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,930 DEBUG [hconnection-0x2d0d5e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,931 DEBUG [hconnection-0x6899296c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,933 DEBUG [hconnection-0x71e7b544-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,935 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,951 DEBUG [hconnection-0x56f5dfc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,954 DEBUG [hconnection-0x71add79-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,955 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,955 DEBUG [hconnection-0x615aaabb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,955 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,955 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,957 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,965 DEBUG [hconnection-0x6743a1f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:57:30,966 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,969 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,983 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:57:30,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:31,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:31,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:31,001 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:31,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:31,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:31,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:31,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,003 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:31,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:31,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:31,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:31,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:31,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:31,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:31,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:31,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:31,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:31,027 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:31,027 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:31,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:31,037 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:57:31,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:31,058 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736311109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736311111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736311111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736311140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736311146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,209 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178c7154eed1d442a7a5fdeb80c2294755 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178c7154eed1d442a7a5fdeb80c2294755 2023-04-17 12:57:31,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736311241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736311248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736311256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736311256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736311242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,317 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/3b4d2e49b05348e0acc4c2d249fc02f1 2023-04-17 12:57:31,421 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/794579ebb960472a994d9a6adbb8410b 2023-04-17 12:57:31,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736311457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736311464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736311471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736311475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736311490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,507 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/1de881f987804cb99ccc746787570534 2023-04-17 12:57:31,552 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/3b4d2e49b05348e0acc4c2d249fc02f1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1 2023-04-17 12:57:31,587 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1, entries=150, sequenceid=18, filesize=25.1 K 2023-04-17 12:57:31,589 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/794579ebb960472a994d9a6adbb8410b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b 2023-04-17 12:57:31,622 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b, entries=150, sequenceid=18, filesize=11.6 K 2023-04-17 12:57:31,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/1de881f987804cb99ccc746787570534 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534 2023-04-17 12:57:31,684 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534, entries=150, sequenceid=18, filesize=11.6 K 2023-04-17 12:57:31,685 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 26c16d52fc9da7952b2d73253571802e in 648ms, sequenceid=18, compaction requested=false 2023-04-17 12:57:31,685 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 12:57:31,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:31,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:31,687 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:31,687 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:31,687 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,687 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:31,687 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:31,699 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:31,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:31,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fba04a8[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:31,701 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:31,701 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,702 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,702 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:31,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:31,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:31,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:31,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:31,712 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:31,715 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:31,715 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:31,715 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:31,715 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:31,715 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:31,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:31,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:31,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:31,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:31,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:31,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:31,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:31,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:31,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:31,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:31,734 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:31,734 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:31,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:31,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:31,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:31,839 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:31,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:31,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736311844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736311911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:31,939 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:31,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:31,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:31,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:31,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,940 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:31,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:31,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:31,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736311935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736311936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:31,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736311936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:31,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:31,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:31,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:32,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:32,007 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:32,017 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,017 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:32,018 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:32,018 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:32,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:32,018 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,026 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:32,026 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:32,029 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,029 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:32,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c011caa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:32,031 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:32,031 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,032 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,032 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:32,032 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:32,032 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:32,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:32,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:32,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:32,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:32,034 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:32,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:32,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:32,035 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:32,035 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:32,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,037 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736312034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,040 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,047 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,047 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,049 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,049 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:32,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:32,049 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:32,049 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:32,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736312049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736312049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736312051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736312052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,059 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f60ae401908b49f094000e883e7c90c1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f60ae401908b49f094000e883e7c90c1 2023-04-17 12:57:32,126 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/e783060c6eee4e93995ebe0d40d065cb 2023-04-17 12:57:32,150 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:32,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:32,174 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:32,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:32,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:32,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,176 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:32,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:32,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:32,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:32,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:32,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:32,228 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,228 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:32,228 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:32,228 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:32,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:32,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,238 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:32,238 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:32,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,239 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,240 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,240 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:32,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@250b189e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:32,241 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:32,241 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:32,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:32,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:32,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:32,244 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:32,245 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:32,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:32,249 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:32,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:32,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:32,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:32,249 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:32,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,251 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,252 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,254 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736312251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,257 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:32,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,257 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,258 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,258 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,258 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,258 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:32,258 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736312259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736312259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736312259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736312264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,342 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:57:32,358 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:32,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:32,454 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:32,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:32,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:32,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,457 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:32,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:32,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:32,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:32,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:32,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:32,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:32,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:32,526 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,526 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:32,526 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:32,526 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:32,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:32,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:32,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:32,545 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,545 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:32,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21895161[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:32,548 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:32,548 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:32,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:32,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:32,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:32,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:32,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:32,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:32,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:32,556 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:32,556 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:32,556 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:32,556 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:32,556 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:32,557 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,557 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,557 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,559 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736312558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,564 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,565 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:32,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,565 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:32,565 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:32,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:32,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736312567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736312567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736312567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:32,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736312561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/c1e542ddffc9408e8a3412c104a343ff 2023-04-17 12:57:32,665 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:32,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:32,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:32,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:32,708 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:32,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:32,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:32,709 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,709 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,710 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,711 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,711 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:32,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:32,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:32,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:32,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:32,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:32,721 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:32,721 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:32,721 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,721 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:32,721 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:32,721 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:32,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:32,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:32,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:32,727 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,727 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,727 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,727 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:32,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e598ebc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:32,729 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:32,729 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:32,734 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:32,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:32,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:32,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:32,735 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:32,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:32,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:32,771 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:32,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:32,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:32,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:32,771 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:32,772 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,772 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,773 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:32,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,778 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:32,778 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:32,878 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:32,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:32,907 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:32,907 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,907 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:32,907 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:32,908 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,908 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,909 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:32,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:32,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:32,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:32,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:32,911 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:32,911 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:32,911 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:32,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:32,927 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:32,927 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:32,927 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:32,927 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:32,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:32,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:32,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:32,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:32,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:32,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a292b91[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:32,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:32,944 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,945 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,945 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:32,950 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:32,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:32,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:32,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:32,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:32,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:32,955 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:32,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:32,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:32,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:32,955 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:32,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:32,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,966 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:32,966 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:32,967 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:32,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:32,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:32,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,967 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:32,967 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:32,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,067 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:33,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:33,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736313067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736313070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:33,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:33,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736313075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:33,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736313075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:33,088 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:33,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:33,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:33,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:33,091 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,091 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,092 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:33,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:33,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:33,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:33,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:33,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736313095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:33,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:33,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:33,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:33,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:33,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:33,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:33,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:33,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b13a2[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:33,105 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:33,105 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,107 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:33,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:33,107 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:33,107 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:33,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:33,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:33,109 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:33,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:33,112 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:33,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:33,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:33,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:33,112 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:33,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/d25d2fa415ed4c41b150c07b42b33cd7 2023-04-17 12:57:33,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,118 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/e783060c6eee4e93995ebe0d40d065cb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb 2023-04-17 12:57:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:33,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:33,123 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:33,123 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:33,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:33,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb, entries=150, sequenceid=38, filesize=25.1 K 2023-04-17 12:57:33,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/c1e542ddffc9408e8a3412c104a343ff as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff 2023-04-17 12:57:33,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff, entries=150, sequenceid=38, filesize=11.6 K 2023-04-17 12:57:33,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/d25d2fa415ed4c41b150c07b42b33cd7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7 2023-04-17 12:57:33,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7, entries=150, sequenceid=38, filesize=11.6 K 2023-04-17 12:57:33,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 26c16d52fc9da7952b2d73253571802e in 1359ms, sequenceid=38, compaction requested=false 2023-04-17 12:57:33,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:33,223 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:33,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:33,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:33,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:33,231 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:33,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:33,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:33,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:33,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,233 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:33,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:33,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:33,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:33,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:33,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:33,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:33,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:33,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:33,235 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:33,235 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:33,235 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:33,236 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:33,331 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179dded1c3b4694aedbca5964396a70427 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179dded1c3b4694aedbca5964396a70427 2023-04-17 12:57:33,363 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/db67b3e01a0a4b99b0812267fb095e0b 2023-04-17 12:57:33,804 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/b36d149810dd47f6b2ec77aa985a45c4 2023-04-17 12:57:33,860 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/a7d1f95857b845f7804140ae55597c28 2023-04-17 12:57:33,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/db67b3e01a0a4b99b0812267fb095e0b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b 2023-04-17 12:57:33,887 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b, entries=150, sequenceid=50, filesize=25.1 K 2023-04-17 12:57:33,889 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/b36d149810dd47f6b2ec77aa985a45c4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4 2023-04-17 12:57:33,897 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 12:57:33,898 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/a7d1f95857b845f7804140ae55597c28 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28 2023-04-17 12:57:33,913 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 12:57:33,914 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 26c16d52fc9da7952b2d73253571802e in 679ms, sequenceid=50, compaction requested=true 2023-04-17 12:57:33,914 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:33,914 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:33,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:33,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:33,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:33,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:33,916 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:33,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c724879[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:33,932 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:33,932 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,933 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:33,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:33,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:33,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:33,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:33,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:33,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:33,938 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:33,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:33,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:33,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:33,938 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:33,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:33,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:33,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:33,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:33,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:33,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:33,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:33,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:33,946 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:33,946 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:33,946 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:33,946 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,046 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,054 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,056 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,070 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,070 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,070 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,078 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,078 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:34,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,088 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,088 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,088 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,094 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efc0725[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,097 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,097 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,098 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,100 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:34,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,103 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,103 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,104 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,104 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,105 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,106 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,108 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,113 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,113 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:34,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:34,127 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:57:34,128 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:34,129 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:34,129 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:34,129 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:34,129 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:34,129 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736254129, max timestamp=1681736254192Num uniques 859; 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736254129, max timestamp=1681736254192Num uniques 859; 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736254129, max timestamp=1681736254192Num uniques 859; 2023-04-17 12:57:34,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:34,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736314194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736314198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736314200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736314206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,213 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736314203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,237 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304170c00a347901a44ed82cbbec84bf964c0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170c00a347901a44ed82cbbec84bf964c0 2023-04-17 12:57:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,244 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,246 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,266 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,266 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,266 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,274 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,274 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,274 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,275 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,275 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:34,275 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,275 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,275 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,275 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,292 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,292 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,293 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3ac8c8[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,297 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,297 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,297 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,303 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,304 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,305 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/94e8c50d44684873a6dd12890fb7db6e 2023-04-17 12:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,319 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,320 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,320 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:34,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736314324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736314324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736314326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736314328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736314336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/09d64be2fd494eca853e005678cecdfe 2023-04-17 12:57:34,420 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,426 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,427 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,427 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,428 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,428 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,428 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,428 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:34,440 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,441 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,442 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,442 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226342ce[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,444 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,444 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,446 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,449 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,449 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,449 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,449 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:34,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,456 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,457 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,459 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,464 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,464 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736314528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736314530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736314539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736314539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736314541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,564 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,581 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,584 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,584 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,594 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,594 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,595 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,595 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,604 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,604 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,604 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:34,604 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,604 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,605 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,605 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,606 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,606 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cace77b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,611 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,611 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,611 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,612 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:34,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,615 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,615 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,615 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,615 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,615 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,616 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,617 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,620 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,620 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,623 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,623 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,624 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,624 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,724 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,727 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,728 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,730 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,730 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,730 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:34,730 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,731 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724706e4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,736 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,737 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,737 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,737 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,737 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:34,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,740 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,740 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,740 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,740 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,740 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,743 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,746 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,746 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,747 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,747 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/c69bdf8f8f264a75bba00e7d52f32b84 2023-04-17 12:57:34,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/94e8c50d44684873a6dd12890fb7db6e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e 2023-04-17 12:57:34,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e, entries=400, sequenceid=62, filesize=59.0 K 2023-04-17 12:57:34,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/09d64be2fd494eca853e005678cecdfe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe 2023-04-17 12:57:34,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 12:57:34,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/c69bdf8f8f264a75bba00e7d52f32b84 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84 2023-04-17 12:57:34,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 12:57:34,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 26c16d52fc9da7952b2d73253571802e in 702ms, sequenceid=62, compaction requested=true 2023-04-17 12:57:34,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:34,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:34,829 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:34,829 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:34,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:34,829 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:34,832 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:34,832 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:34,832 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/B is initiating minor compaction (all files) 2023-04-17 12:57:34,832 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/A is initiating minor compaction (all files) 2023-04-17 12:57:34,832 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/B in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,832 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/A in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,832 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=46.4 K 2023-04-17 12:57:34,832 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=134.5 K 2023-04-17 12:57:34,833 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3b4d2e49b05348e0acc4c2d249fc02f1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681736250986 2023-04-17 12:57:34,833 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 794579ebb960472a994d9a6adbb8410b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681736250986 2023-04-17 12:57:34,833 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e783060c6eee4e93995ebe0d40d065cb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681736251101 2023-04-17 12:57:34,834 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c1e542ddffc9408e8a3412c104a343ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681736251101 2023-04-17 12:57:34,834 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting db67b3e01a0a4b99b0812267fb095e0b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681736251844 2023-04-17 12:57:34,834 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b36d149810dd47f6b2ec77aa985a45c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681736251844 2023-04-17 12:57:34,834 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 94e8c50d44684873a6dd12890fb7db6e, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254099 2023-04-17 12:57:34,835 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 09d64be2fd494eca853e005678cecdfe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254104 2023-04-17 12:57:34,847 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:34,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:34,862 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#B#compaction#123 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:34,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:34,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:34,863 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:34,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:34,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:34,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=167.72 KB heapSize=418.97 KB 2023-04-17 12:57:34,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:34,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:34,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:34,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:34,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:34,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:34,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,871 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,871 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,872 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,872 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:34,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:34,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736314869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736314871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736314872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:34,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:34,891 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:34,891 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:34,891 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:34,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736314873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736314871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,911 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:34,911 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:34,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:34,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:34,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:34,912 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:34,912 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:34,912 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:34,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:34,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:34,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5990fcec[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:34,929 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:34,929 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,930 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:34,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:34,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:34,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:34,932 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:34,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,938 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#A#compaction#124 average throughput is 0.35 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:34,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:34,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:34,942 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:34,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:34,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:34,942 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:34,943 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:34,944 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:34,944 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,944 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:34,945 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:34,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:34,952 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:34,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:34,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:34,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:34,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:34,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:34,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:34,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:34,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:34,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:34,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:34,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:34,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:34,957 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:34,957 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:34,976 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c0668012bbb6416fa0aa8c3a499394a7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c0668012bbb6416fa0aa8c3a499394a7 2023-04-17 12:57:34,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736314984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:34,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:34,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736314984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736314998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736315001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736315001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,057 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,071 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,072 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,074 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,074 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,074 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,074 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,076 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,079 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,079 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@615da022[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,081 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,081 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,084 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,084 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,085 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,088 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,088 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,088 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,089 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,095 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,096 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,096 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,096 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,096 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736315188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736315189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,196 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736315203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,206 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,206 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,210 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,210 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,210 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,210 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,212 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,212 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736315207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736315207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,217 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,217 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af1044d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,225 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,225 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,226 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,227 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,227 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,227 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,232 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,232 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,232 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,232 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,232 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,235 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,235 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,236 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,236 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,238 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,242 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,242 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:35,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,243 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,244 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,330 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/7d995db14144477da65e2d0d06b2db2c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/7d995db14144477da65e2d0d06b2db2c 2023-04-17 12:57:35,337 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/B of 26c16d52fc9da7952b2d73253571802e into 7d995db14144477da65e2d0d06b2db2c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:35,337 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:35,337 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/B, priority=12, startTime=1681736254829; duration=0sec 2023-04-17 12:57:35,337 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:35,337 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:35,339 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:35,339 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/C is initiating minor compaction (all files) 2023-04-17 12:57:35,339 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/C in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,339 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=46.4 K 2023-04-17 12:57:35,340 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 1de881f987804cb99ccc746787570534, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681736250986 2023-04-17 12:57:35,340 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d25d2fa415ed4c41b150c07b42b33cd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681736251101 2023-04-17 12:57:35,340 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a7d1f95857b845f7804140ae55597c28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681736251844 2023-04-17 12:57:35,342 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c69bdf8f8f264a75bba00e7d52f32b84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254104 2023-04-17 12:57:35,346 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,362 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,364 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#C#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:35,369 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,369 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,370 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,372 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,372 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,386 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,386 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,386 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,386 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10cb59d1[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,392 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,393 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,393 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,393 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,393 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,393 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,394 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,394 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,394 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,395 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,401 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=92, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/384e29545a084ec2b42e4adb75f7fd71 2023-04-17 12:57:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,406 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,406 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,408 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,408 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,408 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,425 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,425 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,426 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,426 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/fc677b24298e46e092c2d7be2be84caf 2023-04-17 12:57:35,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736315496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736315500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736315504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736315516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:35,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736315517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,526 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,530 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,531 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,532 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,532 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,535 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,535 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,535 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,535 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,535 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,536 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,536 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,537 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,537 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,537 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,537 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ee2b0b8[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,541 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,541 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,542 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,544 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,548 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,548 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,548 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,548 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,548 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,560 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,561 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,561 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:35,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,562 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,562 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,663 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,666 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,667 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,669 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,669 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,669 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,669 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@423255f6[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,673 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,673 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,673 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,680 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,680 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,683 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,685 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,685 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,786 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,789 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,807 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/29790280e8b34692aa54f9595a43bdf5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/29790280e8b34692aa54f9595a43bdf5 2023-04-17 12:57:35,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:35,815 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:35,821 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/C of 26c16d52fc9da7952b2d73253571802e into 29790280e8b34692aa54f9595a43bdf5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:35,821 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:35,821 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/C, priority=12, startTime=1681736254829; duration=0sec 2023-04-17 12:57:35,822 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1033f291[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:35,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:35,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:35,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:35,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:35,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:35,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:35,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:35,829 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:35,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:35,831 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:35,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:35,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:35,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:35,831 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:35,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,836 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:35,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,839 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:35,850 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,850 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:35,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:35,853 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:35,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:35,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,854 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:35,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,854 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:35,856 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/cd3b1bf625f942d3a5f24adf1b8fa2dc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/cd3b1bf625f942d3a5f24adf1b8fa2dc 2023-04-17 12:57:35,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:35,877 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/A of 26c16d52fc9da7952b2d73253571802e into cd3b1bf625f942d3a5f24adf1b8fa2dc(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 12:57:35,877 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:35,878 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/A, priority=12, startTime=1681736254829; duration=1sec 2023-04-17 12:57:35,878 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:35,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/1458830ca1764432b54c6670c054ad48 2023-04-17 12:57:35,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/384e29545a084ec2b42e4adb75f7fd71 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71 2023-04-17 12:57:35,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71, entries=200, sequenceid=92, filesize=31.9 K 2023-04-17 12:57:35,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/fc677b24298e46e092c2d7be2be84caf as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf 2023-04-17 12:57:35,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf, entries=150, sequenceid=92, filesize=11.6 K 2023-04-17 12:57:35,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/1458830ca1764432b54c6670c054ad48 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48 2023-04-17 12:57:35,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48, entries=150, sequenceid=92, filesize=11.6 K 2023-04-17 12:57:35,954 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:35,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:35,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:35,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~458.77 KB/469776, currentSize=20.13 KB/20610 for 26c16d52fc9da7952b2d73253571802e in 1099ms, sequenceid=92, compaction requested=false 2023-04-17 12:57:35,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:35,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:35,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:35,978 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:35,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:35,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:35,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,979 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:35,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:35,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:35,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:35,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:35,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:35,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:35,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:35,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:35,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:35,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:35,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:35,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:35,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:35,994 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:35,994 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:35,994 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:35,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:36,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:36,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:36,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736256006, max timestamp=1681736256067Num uniques 859; 2023-04-17 12:57:36,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:36,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736256006, max timestamp=1681736256067Num uniques 859; 2023-04-17 12:57:36,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:36,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736256006, max timestamp=1681736256067Num uniques 859; 2023-04-17 12:57:36,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:36,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736316085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736316089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736316089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736316090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736316090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736316191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736316193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736316194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736316195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736316195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736316396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736316402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736316402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736316403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736316404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,478 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417406826f295074c078ff13723f615608c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417406826f295074c078ff13723f615608c 2023-04-17 12:57:36,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736316702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736316705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736316711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736316711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:36,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736316718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:36,898 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/882e0dea612746b3b8b47f54276cfcf8 2023-04-17 12:57:37,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:37,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736317209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:37,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:37,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736317215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736317215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:37,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736317218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:37,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736317219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,365 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/c5d15d666bef4b75bfd691ada429d886 2023-04-17 12:57:37,407 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/f2b7c588fd3e45459f7fd7175034bc25 2023-04-17 12:57:37,415 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/882e0dea612746b3b8b47f54276cfcf8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8 2023-04-17 12:57:37,422 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8, entries=100, sequenceid=101, filesize=18.4 K 2023-04-17 12:57:37,423 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/c5d15d666bef4b75bfd691ada429d886 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886 2023-04-17 12:57:37,428 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886, entries=100, sequenceid=101, filesize=9.3 K 2023-04-17 12:57:37,429 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/f2b7c588fd3e45459f7fd7175034bc25 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25 2023-04-17 12:57:37,434 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25, entries=100, sequenceid=101, filesize=9.3 K 2023-04-17 12:57:37,435 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=181.14 KB/185490 for 26c16d52fc9da7952b2d73253571802e in 1441ms, sequenceid=101, compaction requested=true 2023-04-17 12:57:37,436 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:37,436 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:37,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:37,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:37,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:37,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:37,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:37,440 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:37,441 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:37,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7c2497[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:37,441 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:37,442 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:37,443 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:37,443 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:37,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:37,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:37,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:37,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:37,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:37,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:37,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,465 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:37,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:37,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:37,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:37,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:37,467 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:37,467 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:37,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:37,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:37,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:37,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:37,568 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:37,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:37,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:37,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:37,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:37,573 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:37,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:37,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:37,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:37,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:37,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:37,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:37,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:37,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:37,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:37,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:37,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:37,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:37,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:37,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:37,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:37,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:37,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:37,576 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:37,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:37,577 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=181.14 KB heapSize=458.81 KB 2023-04-17 12:57:37,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:37,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:37,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:37,578 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:37,578 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:37,578 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:38,016 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417486391187fd741078df36d8a0ff1421c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417486391187fd741078df36d8a0ff1421c 2023-04-17 12:57:38,028 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=60.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/61d126e6abca457faa68d3ef47280814 2023-04-17 12:57:38,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:38,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:38,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736318227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736318232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736318236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736318237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736318238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736318340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736318347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736318347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736318348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,462 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a 2023-04-17 12:57:38,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736318548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736318553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736318569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736318569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736318852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736318860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736318872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736318872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,888 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/c8708179e88544baa161f34886f97223 2023-04-17 12:57:38,892 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/61d126e6abca457faa68d3ef47280814 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814 2023-04-17 12:57:38,897 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814, entries=150, sequenceid=131, filesize=25.2 K 2023-04-17 12:57:38,898 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a 2023-04-17 12:57:38,912 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a, entries=150, sequenceid=131, filesize=11.7 K 2023-04-17 12:57:38,913 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/c8708179e88544baa161f34886f97223 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223 2023-04-17 12:57:38,929 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223, entries=150, sequenceid=131, filesize=11.7 K 2023-04-17 12:57:38,930 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~458.77 KB/469776, currentSize=26.84 KB/27480 for 26c16d52fc9da7952b2d73253571802e in 1353ms, sequenceid=131, compaction requested=true 2023-04-17 12:57:38,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:38,930 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:38,930 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:38,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:38,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:38,931 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:38,932 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:38,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,932 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:38,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:38,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1403810[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:38,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:38,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:38,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:38,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:38,936 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,936 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:38,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:38,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:38,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:38,937 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:38,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:38,939 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:38,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:38,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:38,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:38,939 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:38,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:38,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:38,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:38,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:38,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:38,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:38,945 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:38,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:38,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:38,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:38,945 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:38,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:38,946 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:38,946 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:38,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:39,046 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:39,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:39,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:39,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:39,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:39,069 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:39,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:39,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:39,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:39,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:39,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:39,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:39,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:39,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:39,082 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:39,082 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:39,082 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,082 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:39,082 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:39,083 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:39,132 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f89d38969b2f4d81af7613e3191c5ada to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f89d38969b2f4d81af7613e3191c5ada 2023-04-17 12:57:39,163 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/16f33a9e49ce4a949e8fd048f6a34d73 2023-04-17 12:57:39,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:39,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:39,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736259375, max timestamp=1681736259442Num uniques 859; 2023-04-17 12:57:39,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736259375, max timestamp=1681736259442Num uniques 859; 2023-04-17 12:57:39,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:39,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:39,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736259375, max timestamp=1681736259442Num uniques 859; 2023-04-17 12:57:39,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:39,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736319483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736319484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736319485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736319485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736319587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736319591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736319593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736319594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,613 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/87f94af240514e64a861334c741ac321 2023-04-17 12:57:39,661 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/bf4d97410151490f9554d840e03477cc 2023-04-17 12:57:39,666 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/16f33a9e49ce4a949e8fd048f6a34d73 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73 2023-04-17 12:57:39,675 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73, entries=100, sequenceid=138, filesize=18.5 K 2023-04-17 12:57:39,677 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/87f94af240514e64a861334c741ac321 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321 2023-04-17 12:57:39,682 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321, entries=100, sequenceid=138, filesize=9.4 K 2023-04-17 12:57:39,686 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/bf4d97410151490f9554d840e03477cc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc 2023-04-17 12:57:39,691 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc, entries=100, sequenceid=138, filesize=9.4 K 2023-04-17 12:57:39,693 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=181.14 KB/185490 for 26c16d52fc9da7952b2d73253571802e in 610ms, sequenceid=138, compaction requested=true 2023-04-17 12:57:39,693 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:39,693 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:39,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:39,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:39,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:39,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,697 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:39,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:39,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:39,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518d402b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:39,700 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:39,700 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,701 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:39,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:39,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:39,703 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:39,703 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:39,703 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:39,704 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,704 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,704 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:39,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:39,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:39,708 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:39,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:39,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:39,708 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:39,708 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:39,711 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:39,712 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,712 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:39,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:39,712 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:39,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,713 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:39,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:39,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:39,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,720 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,720 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:39,720 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:39,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:39,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:39,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:39,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:39,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=187.85 KB heapSize=478.73 KB 2023-04-17 12:57:39,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:39,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:39,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:39,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:39,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:39,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736319796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736319798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736319798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736319799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,820 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:39,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:39,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:39,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:39,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:39,832 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:39,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:39,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:39,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:39,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:39,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:39,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:39,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:39,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:39,837 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:39,837 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,837 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:39,846 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:39,846 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:39,858 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,858 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:39,858 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:39,858 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:39,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:39,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:39,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:39,859 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,859 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:39,860 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,860 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:39,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:39,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:39,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:39,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea597f1[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:39,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,863 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,863 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:39,863 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:39,863 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:39,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:39,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:39,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:39,869 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:39,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:39,870 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:39,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:39,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:39,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:39,870 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:39,871 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,871 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:39,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,871 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,872 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:39,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,873 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:39,879 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:39,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:39,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:39,880 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:39,880 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:39,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736319900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:39,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736319900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,981 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:39,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:39,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:39,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:39,986 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:39,986 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,986 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:39,986 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:39,986 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,986 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,987 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:39,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:39,987 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:39,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:39,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:39,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:39,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:39,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:39,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:39,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:39,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:39,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:39,994 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:39,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:39,995 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:39,995 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:39,995 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:39,995 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,995 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:39,995 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:39,996 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,996 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:39,996 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:39,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6546482e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,011 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,011 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,011 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,015 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,016 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,018 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,018 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,019 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,020 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,027 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,029 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,030 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,031 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,031 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,031 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:40,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736320102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736320111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736320111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736320111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,131 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,157 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,161 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,161 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,163 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,163 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,164 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,172 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,172 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:40,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:40,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:40,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:40,174 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c67c2e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,179 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,179 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,182 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,185 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,189 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,189 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,191 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,191 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,194 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,195 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,197 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,197 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:40,235 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c29651ac6ddd40d58f623acbf7e47584 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c29651ac6ddd40d58f623acbf7e47584 2023-04-17 12:57:40,243 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/9f4ff4378c854566973aafe55de5d0a3 2023-04-17 12:57:40,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736320251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,263 DEBUG [Thread-834] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:40,297 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,330 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,331 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,332 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,332 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:40,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:40,337 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,337 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,337 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:40,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e71e4a5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,342 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/164a595b3cef49f59ec4f23d18aa4640 2023-04-17 12:57:40,343 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,343 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,350 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,351 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,352 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,356 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,357 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,359 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,359 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:40,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736320417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736320419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,459 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,462 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,483 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,483 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,483 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,488 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,488 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,488 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,488 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,489 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,489 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:40,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:40,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:40,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:40,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,491 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,491 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c99ea85[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,492 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,492 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,493 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,493 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,493 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,493 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,493 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,493 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,513 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,513 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,517 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,519 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,521 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,521 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,522 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:40,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736320608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,622 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:40,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736320620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,626 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,626 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,626 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,626 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,629 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,630 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,630 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,630 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,630 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:40,630 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:40,632 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc9c000[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,636 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,637 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,637 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,640 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,640 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,640 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,649 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,649 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,649 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,749 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,755 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,763 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,763 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,763 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,764 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,764 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,764 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,764 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:40,764 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:40,765 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:40,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db8f45b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:40,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:40,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,770 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:40,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:40,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:40,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:40,772 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,772 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,772 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,772 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:40,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:40,774 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:40,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:40,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:40,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:40,774 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:40,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:40,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,776 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:40,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:40,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/23c81af2806d4417904fa200746107c9 2023-04-17 12:57:40,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:40,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:40,796 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:40,796 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:40,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/9f4ff4378c854566973aafe55de5d0a3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3 2023-04-17 12:57:40,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3, entries=250, sequenceid=170, filesize=38.9 K 2023-04-17 12:57:40,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/164a595b3cef49f59ec4f23d18aa4640 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640 2023-04-17 12:57:40,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 12:57:40,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/23c81af2806d4417904fa200746107c9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9 2023-04-17 12:57:40,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9, entries=150, sequenceid=170, filesize=11.8 K 2023-04-17 12:57:40,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~498.61 KB/510576, currentSize=6.71 KB/6870 for 26c16d52fc9da7952b2d73253571802e in 1050ms, sequenceid=170, compaction requested=true 2023-04-17 12:57:40,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:40,845 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:40,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:40,846 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:40,846 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:40,846 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:57:40,851 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 161992 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:40,851 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/A is initiating minor compaction (all files) 2023-04-17 12:57:40,851 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/A in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,851 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/cd3b1bf625f942d3a5f24adf1b8fa2dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=158.2 K 2023-04-17 12:57:40,852 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67116 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:40,852 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/B is initiating minor compaction (all files) 2023-04-17 12:57:40,852 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting cd3b1bf625f942d3a5f24adf1b8fa2dc, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254104 2023-04-17 12:57:40,852 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/B in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,853 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/7d995db14144477da65e2d0d06b2db2c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=65.5 K 2023-04-17 12:57:40,853 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 384e29545a084ec2b42e4adb75f7fd71, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681736254200 2023-04-17 12:57:40,853 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7d995db14144477da65e2d0d06b2db2c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254104 2023-04-17 12:57:40,854 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 882e0dea612746b3b8b47f54276cfcf8, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681736254869 2023-04-17 12:57:40,854 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting fc677b24298e46e092c2d7be2be84caf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681736254200 2023-04-17 12:57:40,855 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 61d126e6abca457faa68d3ef47280814, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736256087 2023-04-17 12:57:40,855 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c5d15d666bef4b75bfd691ada429d886, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681736254869 2023-04-17 12:57:40,856 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 16f33a9e49ce4a949e8fd048f6a34d73, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681736258234 2023-04-17 12:57:40,856 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3eb15cd11f9a4a5abd9b41b5bc4ff25a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736256087 2023-04-17 12:57:40,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 87f94af240514e64a861334c741ac321, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681736258234 2023-04-17 12:57:40,858 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9f4ff4378c854566973aafe55de5d0a3, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259481 2023-04-17 12:57:40,858 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 164a595b3cef49f59ec4f23d18aa4640, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259483 2023-04-17 12:57:40,880 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#B#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:40,895 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#A#compaction#142 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:40,897 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:40,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:40,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:40,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:40,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:40,909 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:40,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:40,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:40,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,909 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,910 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:40,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:40,911 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:40,911 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:40,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:40,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:40,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:40,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:40,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:40,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:40,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:40,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:40,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:40,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:40,915 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:40,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:40,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:40,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:41,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736260931, max timestamp=1681736261010Num uniques 859; 2023-04-17 12:57:41,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736260931, max timestamp=1681736261010Num uniques 859; 2023-04-17 12:57:41,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:41,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681736260931, max timestamp=1681736261010Num uniques 859; 2023-04-17 12:57:41,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:41,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 12:57:41,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736321037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736321038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736321146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736321146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,316 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/e28a71d230cd453f9d5e67406604df7d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/e28a71d230cd453f9d5e67406604df7d 2023-04-17 12:57:41,329 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/B of 26c16d52fc9da7952b2d73253571802e into e28a71d230cd453f9d5e67406604df7d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:41,329 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:41,329 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/B, priority=10, startTime=1681736260846; duration=0sec 2023-04-17 12:57:41,329 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:41,329 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:41,337 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67116 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:41,337 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/C is initiating minor compaction (all files) 2023-04-17 12:57:41,338 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/C in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:41,338 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/29790280e8b34692aa54f9595a43bdf5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=65.5 K 2023-04-17 12:57:41,339 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 29790280e8b34692aa54f9595a43bdf5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681736254104 2023-04-17 12:57:41,340 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1458830ca1764432b54c6670c054ad48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681736254200 2023-04-17 12:57:41,340 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f2b7c588fd3e45459f7fd7175034bc25, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681736254869 2023-04-17 12:57:41,341 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c8708179e88544baa161f34886f97223, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736256087 2023-04-17 12:57:41,346 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting bf4d97410151490f9554d840e03477cc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681736258234 2023-04-17 12:57:41,346 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 23c81af2806d4417904fa200746107c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259483 2023-04-17 12:57:41,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736321353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736321355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,360 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417eeb98b15c46b4cd38b41b8361fa51575 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eeb98b15c46b4cd38b41b8361fa51575 2023-04-17 12:57:41,372 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/4d6113645fa348f0800a12e73779cb40 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/4d6113645fa348f0800a12e73779cb40 2023-04-17 12:57:41,380 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#C#compaction#144 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:41,381 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/A of 26c16d52fc9da7952b2d73253571802e into 4d6113645fa348f0800a12e73779cb40(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:41,381 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:41,381 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/A, priority=10, startTime=1681736260845; duration=0sec 2023-04-17 12:57:41,382 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:41,402 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/c5fdd11b66ad481c95bd6d9058c6fd3f 2023-04-17 12:57:41,429 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/e44f80dae1fd4aee8f4c0f4aca7f049a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/e44f80dae1fd4aee8f4c0f4aca7f049a 2023-04-17 12:57:41,442 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/C of 26c16d52fc9da7952b2d73253571802e into e44f80dae1fd4aee8f4c0f4aca7f049a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:41,442 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:41,443 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/C, priority=10, startTime=1681736260846; duration=0sec 2023-04-17 12:57:41,443 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:41,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736321617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736321632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736321656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:41,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736321658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:41,856 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/3ab409f0bbb34de7aabf421beed01976 2023-04-17 12:57:42,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33250 deadline: 1681736322160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33230 deadline: 1681736322161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,271 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/ef988e3a57f14f9e847ed4e13252934e 2023-04-17 12:57:42,278 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/c5fdd11b66ad481c95bd6d9058c6fd3f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f 2023-04-17 12:57:42,284 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f, entries=50, sequenceid=174, filesize=11.6 K 2023-04-17 12:57:42,285 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/3ab409f0bbb34de7aabf421beed01976 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976 2023-04-17 12:57:42,293 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976, entries=50, sequenceid=174, filesize=7.1 K 2023-04-17 12:57:42,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/ef988e3a57f14f9e847ed4e13252934e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e 2023-04-17 12:57:42,303 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e, entries=50, sequenceid=174, filesize=7.1 K 2023-04-17 12:57:42,304 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=194.56 KB/199230 for 26c16d52fc9da7952b2d73253571802e in 1389ms, sequenceid=174, compaction requested=false 2023-04-17 12:57:42,304 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:42,304 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:42,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:42,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:42,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:42,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,305 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,306 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,306 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:42,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3d7985[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:42,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:42,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,311 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:42,311 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:42,311 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:42,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:42,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:42,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:42,312 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:42,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:42,315 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:42,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:42,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:42,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:42,315 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:42,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,318 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:42,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:42,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,326 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:42,326 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:42,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:42,426 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:42,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:42,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:42,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:42,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:42,431 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:42,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:42,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:42,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,432 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:42,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:42,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:42,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:42,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:42,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:42,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:42,434 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,434 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:42,434 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=194.56 KB heapSize=498.66 KB 2023-04-17 12:57:42,434 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:42,434 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:42,434 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:42,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:42,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:42,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:42,462 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417466b6d9154e645f685b81aaa09072333 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417466b6d9154e645f685b81aaa09072333 2023-04-17 12:57:42,467 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/7f00ff30c01c4b3cb10aba1785de165b 2023-04-17 12:57:42,502 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/1b9e0c0d507344b5b0c65687a22e4999 2023-04-17 12:57:42,518 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/97a41f60da4e4d6a8bd68dfa870a7b0a 2023-04-17 12:57:42,522 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/7f00ff30c01c4b3cb10aba1785de165b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b 2023-04-17 12:57:42,536 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b, entries=150, sequenceid=209, filesize=25.3 K 2023-04-17 12:57:42,537 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/1b9e0c0d507344b5b0c65687a22e4999 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999 2023-04-17 12:57:42,544 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999, entries=150, sequenceid=209, filesize=11.8 K 2023-04-17 12:57:42,545 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/97a41f60da4e4d6a8bd68dfa870a7b0a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a 2023-04-17 12:57:42,574 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a, entries=150, sequenceid=209, filesize=11.8 K 2023-04-17 12:57:42,575 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~498.61 KB/510576, currentSize=0 B/0 for 26c16d52fc9da7952b2d73253571802e in 141ms, sequenceid=209, compaction requested=true 2023-04-17 12:57:42,575 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:42,575 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:42,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:42,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:42,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:42,576 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,579 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,579 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:42,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f937945[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:42,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:42,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:42,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:42,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:42,587 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:42,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:42,588 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:42,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:42,592 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:42,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:42,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:42,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:42,592 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:42,595 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,596 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,596 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,597 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:42,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:42,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,619 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,619 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,719 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:42,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:42,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:42,738 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:42,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:42,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:42,739 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,739 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,739 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,739 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,740 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,740 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:42,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:42,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:42,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:42,766 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:42,766 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:42,766 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,766 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:42,766 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:42,767 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:42,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:42,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:42,767 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:42,768 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,771 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,773 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:42,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aef7377[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:42,775 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:42,775 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,776 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,776 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:42,776 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:42,776 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:42,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:42,777 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:42,777 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,777 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,777 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:42,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:42,789 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:42,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:42,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:42,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:42,790 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:42,792 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,793 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,793 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,793 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:42,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:42,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:42,800 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:42,801 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:42,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,901 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:42,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:42,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:42,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:42,910 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:42,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:42,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:42,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,915 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:42,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:42,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,926 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,926 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:42,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:42,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:42,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:42,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:42,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:42,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:42,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:42,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:42,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:42,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:42,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:42,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@409f505d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:42,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:42,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:42,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:42,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:42,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:42,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,957 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:42,957 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:42,957 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:42,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:42,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:42,978 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:42,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:42,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:42,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:42,979 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:42,989 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,989 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:42,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:42,998 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,008 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,009 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,009 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,109 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:43,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:43,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:43,135 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:43,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:43,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:43,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,136 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,137 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,137 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,137 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:43,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:43,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:43,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:43,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:43,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:43,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:43,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:43,166 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,166 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:43,166 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:43,166 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:43,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:43,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:43,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:43,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,168 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:43,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f9e89a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:43,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:43,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,171 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,171 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,171 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:43,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:43,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:43,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:43,172 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:43,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:43,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:43,178 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:43,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:43,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:43,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:43,178 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:43,179 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,179 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,179 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,187 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,187 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:43,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:43,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:43,292 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:43,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:43,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:43,304 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:43,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:43,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:43,305 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,305 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:43,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:43,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:43,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:43,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:43,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:43,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:43,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:43,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:43,335 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:43,335 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:43,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:43,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:43,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:43,336 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,337 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,337 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5324ea71[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:43,339 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:43,339 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,340 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:43,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:43,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:43,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:43,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:43,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:43,342 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:43,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:43,345 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:43,345 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:43,345 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:43,345 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:43,345 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:43,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,348 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,349 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,351 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,353 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,353 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,355 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,355 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,355 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681736263318, max timestamp=1681736263354Num uniques 859; 2023-04-17 12:57:43,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-17 12:57:43,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736263340, max timestamp=1681736263354Num uniques 859; 2023-04-17 12:57:43,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:43,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736263340, max timestamp=1681736263354Num uniques 859; 2023-04-17 12:57:43,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:57:43,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736323416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736323416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,455 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:43,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:43,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:43,464 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:43,464 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,464 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:43,464 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:43,468 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,473 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:43,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:43,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:43,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:43,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:43,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:43,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:43,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:43,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:43,483 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:43,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:43,496 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:43,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2814fa9[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:43,499 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:43,499 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,500 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:43,500 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:43,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:43,503 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,503 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:43,503 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,504 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:43,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:43,507 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:43,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:43,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:43,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:43,507 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:43,513 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,513 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,514 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,518 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,523 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,526 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,527 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736323522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736323532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,627 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:43,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736323627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,664 DEBUG [Thread-830] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:43,674 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:43,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:43,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:43,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736323671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,683 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,683 DEBUG [Thread-828] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:43,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:43,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:43,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:43,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:43,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:43,713 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:43,713 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,713 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:43,714 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:43,714 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:43,714 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,714 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:43,714 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:43,714 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,714 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:43,714 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:43,716 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,717 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:43,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38a6ea1c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:43,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:43,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,721 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:43,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:43,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:43,722 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,722 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:43,723 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,723 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,723 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,723 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,732 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e22e12a9483149a4a05131168f139cb1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e22e12a9483149a4a05131168f139cb1 2023-04-17 12:57:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:43,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:43,733 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:43,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:43,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:43,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:43,733 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:43,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,736 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,735 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,736 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:43,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736323740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736323731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,736 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,748 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,749 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,749 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:43,771 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=222, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/d8b5c942e4ce4c9691582150ce06e557 2023-04-17 12:57:43,850 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:43,855 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:43,855 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,855 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:43,855 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:43,860 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,860 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,862 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:43,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:43,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:43,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:43,867 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:43,867 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:43,867 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:43,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:43,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:43,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:43,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:43,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:43,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,878 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,879 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,879 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:43,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@208368aa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:43,880 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:43,880 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,881 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,881 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:43,881 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:43,881 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:43,881 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,881 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:43,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:43,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:43,882 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:43,884 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:43,884 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:43,884 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:43,884 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:43,884 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:43,885 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,885 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,885 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,887 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:43,894 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,896 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:43,896 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:43,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:43,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:43,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:43,896 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:43,897 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:43,897 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:43,896 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:43,897 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:43,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:43,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:43,997 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:43,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:44,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:44,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:44,000 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:44,000 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,000 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:44,000 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:44,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,003 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,003 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:44,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:44,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:44,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:44,008 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,008 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:44,008 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:44,008 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:44,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:44,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:44,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@540069a9[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:44,012 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:44,012 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,014 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:44,014 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:44,014 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:44,015 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,015 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:44,015 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:44,015 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,015 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:44,017 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:44,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:44,020 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:44,020 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:44,020 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:44,020 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:44,020 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:44,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,022 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,024 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,031 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,031 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,032 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,034 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,034 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:44,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:44,034 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,034 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,035 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,035 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,035 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,035 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:44,035 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:44,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:44,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736324048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736324050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,138 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:44,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:44,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:44,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:44,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:44,146 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:44,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:44,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:44,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,149 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,150 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,150 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:44,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:44,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:44,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:44,158 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:44,158 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,158 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:44,160 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:44,160 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:44,161 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,161 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:44,161 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:44,161 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:44,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:44,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:44,162 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:44,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:44,164 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:44,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf49887[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:44,167 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:44,167 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:44,168 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:44,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:44,168 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:44,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:44,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:44,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:44,175 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:44,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:44,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:44,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:44,175 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:44,178 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,178 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,178 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,185 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:44,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:44,186 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:44,186 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:44,186 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,186 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/1f5517275d9c45bd86660c1100035f85 2023-04-17 12:57:44,286 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:44,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:44,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:44,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4 2023-04-17 12:57:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:44,308 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:44,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:44,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:44,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:44,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33214 deadline: 1681736324307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,317 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,319 DEBUG [Thread-834] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8227 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:44,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:44,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:44,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:44,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/d8b5c942e4ce4c9691582150ce06e557 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557 2023-04-17 12:57:44,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:44,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:44,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:44,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:44,334 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:44,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:44,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:44,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:44,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,336 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,338 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,338 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:44,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530d6f0f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:44,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:44,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557, entries=350, sequenceid=222, filesize=52.6 K 2023-04-17 12:57:44,344 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,344 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:44,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/1f5517275d9c45bd86660c1100035f85 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85 2023-04-17 12:57:44,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,346 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:44,346 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:44,346 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:44,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:44,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85, entries=100, sequenceid=222, filesize=9.4 K 2023-04-17 12:57:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:44,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:44,354 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:44,354 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:44,354 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:44,354 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:44,354 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:44,355 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,355 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4 2023-04-17 12:57:44,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,362 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:44,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4, entries=100, sequenceid=222, filesize=9.4 K 2023-04-17 12:57:44,367 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,367 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:44,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:44,368 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,368 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:44,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:44,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:44,369 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:44,369 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:44,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:44,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for 26c16d52fc9da7952b2d73253571802e in 1108ms, sequenceid=222, compaction requested=true 2023-04-17 12:57:44,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:44,371 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:44,371 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:44,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:44,372 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 41306 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:44,373 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/B is initiating minor compaction (all files) 2023-04-17 12:57:44,373 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/B in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,373 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/e28a71d230cd453f9d5e67406604df7d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=40.3 K 2023-04-17 12:57:44,373 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117896 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:44,373 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/A is initiating minor compaction (all files) 2023-04-17 12:57:44,373 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/A in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,373 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/4d6113645fa348f0800a12e73779cb40, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=115.1 K 2023-04-17 12:57:44,373 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e28a71d230cd453f9d5e67406604df7d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259483 2023-04-17 12:57:44,374 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4d6113645fa348f0800a12e73779cb40, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259483 2023-04-17 12:57:44,374 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c5fdd11b66ad481c95bd6d9058c6fd3f, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681736259796 2023-04-17 12:57:44,374 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3ab409f0bbb34de7aabf421beed01976, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681736259796 2023-04-17 12:57:44,374 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7f00ff30c01c4b3cb10aba1785de165b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681736261021 2023-04-17 12:57:44,375 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 1b9e0c0d507344b5b0c65687a22e4999, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681736261021 2023-04-17 12:57:44,375 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d8b5c942e4ce4c9691582150ce06e557, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736263209 2023-04-17 12:57:44,376 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 1f5517275d9c45bd86660c1100035f85, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736263247 2023-04-17 12:57:44,397 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#B#compaction#154 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:44,427 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#A#compaction#153 average throughput is 0.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:44,469 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:44,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:44,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:44,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:44,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:44,476 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:44,476 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,476 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:44,476 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:44,477 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,477 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,477 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:44,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:44,477 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:44,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:44,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:44,485 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/a7bdf258cd004d3aae52335cb60e127f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/a7bdf258cd004d3aae52335cb60e127f 2023-04-17 12:57:44,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:44,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:44,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:44,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:44,491 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:44,491 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:44,491 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,491 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:44,491 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-04-17 12:57:44,491 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:44,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:44,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:44,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:44,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:44,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:44,501 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/B of 26c16d52fc9da7952b2d73253571802e into a7bdf258cd004d3aae52335cb60e127f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:44,501 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:44,501 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/B, priority=12, startTime=1681736264371; duration=0sec 2023-04-17 12:57:44,501 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:44,501 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:57:44,504 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 41306 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:57:44,504 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/C is initiating minor compaction (all files) 2023-04-17 12:57:44,504 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/C in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:44,504 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/e44f80dae1fd4aee8f4c0f4aca7f049a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=40.3 K 2023-04-17 12:57:44,505 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e44f80dae1fd4aee8f4c0f4aca7f049a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681736259483 2023-04-17 12:57:44,507 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ef988e3a57f14f9e847ed4e13252934e, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681736259796 2023-04-17 12:57:44,515 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 97a41f60da4e4d6a8bd68dfa870a7b0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681736261021 2023-04-17 12:57:44,515 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8378a82a4a3f4eb1ac47d58d9e1fc7f4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736263247 2023-04-17 12:57:44,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:44,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:44,571 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#C#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:44,593 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173021bff5381f489cad3584883ccc5230 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173021bff5381f489cad3584883ccc5230 2023-04-17 12:57:44,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736324614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736324619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736324721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736324733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,750 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/ff39155b260242dab2bf72f5a51361dc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ff39155b260242dab2bf72f5a51361dc 2023-04-17 12:57:44,798 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/A of 26c16d52fc9da7952b2d73253571802e into ff39155b260242dab2bf72f5a51361dc(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:44,798 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:44,798 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/A, priority=12, startTime=1681736264371; duration=0sec 2023-04-17 12:57:44,798 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:44,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736324929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:44,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:44,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736324937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,104 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/108d88d5a8e0413b921059e5b402af3e 2023-04-17 12:57:45,107 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/4337044f60974dc28be9cfa5f8810fc2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/4337044f60974dc28be9cfa5f8810fc2 2023-04-17 12:57:45,127 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/C of 26c16d52fc9da7952b2d73253571802e into 4337044f60974dc28be9cfa5f8810fc2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:45,128 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:45,128 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/C, priority=12, startTime=1681736264371; duration=0sec 2023-04-17 12:57:45,128 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:45,147 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ef12b7c6f05147d38f2c53a6a0accd53 2023-04-17 12:57:45,206 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/47039e06fcbd4042bc3c548192e78917 2023-04-17 12:57:45,211 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/108d88d5a8e0413b921059e5b402af3e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e 2023-04-17 12:57:45,235 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e, entries=150, sequenceid=259, filesize=25.4 K 2023-04-17 12:57:45,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:45,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736325235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,239 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ef12b7c6f05147d38f2c53a6a0accd53 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53 2023-04-17 12:57:45,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736325239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,265 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53, entries=150, sequenceid=259, filesize=11.9 K 2023-04-17 12:57:45,271 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/47039e06fcbd4042bc3c548192e78917 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917 2023-04-17 12:57:45,288 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917, entries=150, sequenceid=259, filesize=11.9 K 2023-04-17 12:57:45,290 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~405.25 KB/414976, currentSize=40.25 KB/41220 for 26c16d52fc9da7952b2d73253571802e in 798ms, sequenceid=259, compaction requested=false 2023-04-17 12:57:45,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:45,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:45,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:45,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:45,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:45,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:45,292 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,292 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:45,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:45,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:45,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68709bfc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:45,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:45,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,295 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:45,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:45,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:45,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:45,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:45,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:45,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:45,296 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:45,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:45,300 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:45,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:45,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:45,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:45,300 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:45,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:45,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:45,307 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:45,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:45,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:45,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:45,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:45,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:45,311 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:45,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:45,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:45,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:45,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:45,314 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:45,314 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:45,418 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:45,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:45,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:45,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:45,430 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:45,430 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:45,430 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:45,430 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:45,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:45,432 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:45,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:45,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:45,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:45,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:45,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:45,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:45,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:45,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:45,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:45,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:45,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:45,435 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:45,435 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:45,436 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:45,436 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:45,489 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417dd4f81588bd942219bdf03852fb180bc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dd4f81588bd942219bdf03852fb180bc 2023-04-17 12:57:45,523 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/68f1194b03094a9ea253d5c9a7937be4 2023-04-17 12:57:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:45,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:45,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736265782, max timestamp=1681736265809Num uniques 859; 2023-04-17 12:57:45,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:45,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736265782, max timestamp=1681736265809Num uniques 859; 2023-04-17 12:57:45,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:45,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736265782, max timestamp=1681736265809Num uniques 859; 2023-04-17 12:57:45,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:57:45,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:45,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736325884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:45,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736325886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:45,966 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ed22039b730c4c6e920f57d679890aff 2023-04-17 12:57:45,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:45,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736325994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736326000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:46,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736326199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736326214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,464 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5f92fee0e09f4b8f89f69b21f2c29250 2023-04-17 12:57:46,502 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/68f1194b03094a9ea253d5c9a7937be4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4 2023-04-17 12:57:46,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:46,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736326505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736326527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,536 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4, entries=150, sequenceid=271, filesize=25.4 K 2023-04-17 12:57:46,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ed22039b730c4c6e920f57d679890aff as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff 2023-04-17 12:57:46,559 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 12:57:46,562 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5f92fee0e09f4b8f89f69b21f2c29250 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250 2023-04-17 12:57:46,568 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 12:57:46,570 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 26c16d52fc9da7952b2d73253571802e in 1135ms, sequenceid=271, compaction requested=true 2023-04-17 12:57:46,570 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:46,570 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:46,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:46,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:46,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:46,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:46,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:46,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:46,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f6aea7a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:46,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:46,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,574 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:46,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:46,576 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:46,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:46,576 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:46,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:46,579 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:46,579 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:46,579 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:46,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:46,580 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:46,581 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,581 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:46,581 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:46,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:46,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:46,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,594 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:46,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:46,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:46,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:46,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:46,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:46,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:46,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:46,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:46,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:46,628 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:46,629 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:46,729 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:46,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:46,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:46,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:46,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:46,734 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:46,734 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:46,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:46,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:46,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:46,748 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:46,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:46,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:46,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:46,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:46,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:46,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:46,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:46,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:46,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:46,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:46,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:46,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:46,751 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:46,751 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:46,752 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:46,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:57:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:47,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:47,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736327046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736327046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736327157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736327159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,203 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c961b27ede4941ab9c2a9c3981559e3e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c961b27ede4941ab9c2a9c3981559e3e 2023-04-17 12:57:47,250 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/fcaa8740896b497e81d6a66b5bf7a1ed 2023-04-17 12:57:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736327367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736327374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33226 deadline: 1681736327674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,685 DEBUG [Thread-830] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:47,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736327681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736327688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,691 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ea7e1c6e05c1477aa148d1c4fc15091c 2023-04-17 12:57:47,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:47,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33184 deadline: 1681736327704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,707 DEBUG [Thread-828] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:57:47,713 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/93edb30d06ec40b380094b50f994b0aa 2023-04-17 12:57:47,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/fcaa8740896b497e81d6a66b5bf7a1ed as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed 2023-04-17 12:57:47,751 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed, entries=150, sequenceid=307, filesize=25.4 K 2023-04-17 12:57:47,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/ea7e1c6e05c1477aa148d1c4fc15091c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c 2023-04-17 12:57:47,763 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c, entries=150, sequenceid=307, filesize=11.9 K 2023-04-17 12:57:47,764 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/93edb30d06ec40b380094b50f994b0aa as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa 2023-04-17 12:57:47,790 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa, entries=150, sequenceid=307, filesize=11.9 K 2023-04-17 12:57:47,794 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 26c16d52fc9da7952b2d73253571802e in 1043ms, sequenceid=307, compaction requested=true 2023-04-17 12:57:47,794 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:47,794 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:47,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:47,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:47,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:47,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:47,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:47,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:47,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47a26db0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:47,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:47,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,807 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,807 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:47,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:47,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:47,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:47,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:47,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:47,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:47,820 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:47,820 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:47,820 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:47,821 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:47,821 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:47,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,828 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:47,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:47,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:47,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:47,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:47,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:47,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:47,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:47,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:47,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:47,839 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:47,839 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:47,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:47,939 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:47,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:47,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:47,962 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:47,962 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:47,962 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:47,962 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:47,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:47,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,964 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:47,964 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:47,964 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:47,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:47,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:47,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:47,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:47,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:47,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:47,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:47,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:47,988 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:47,988 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:47,988 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:47,988 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:47,988 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 12:57:47,988 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:47,988 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:47,988 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:47,989 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:47,989 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:47,989 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:48,051 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171dbbc6e94c1f4249987e04a3ffe42529 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171dbbc6e94c1f4249987e04a3ffe42529 2023-04-17 12:57:48,092 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/8ea54e5907d34aee98f16a3321acaeda 2023-04-17 12:57:48,157 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/4358d499c58d4eebb6b2136fd3448337 2023-04-17 12:57:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:48,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:48,234 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/112b856f99e244369824dce8aeef3f78 2023-04-17 12:57:48,240 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/8ea54e5907d34aee98f16a3321acaeda as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda 2023-04-17 12:57:48,245 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda, entries=100, sequenceid=316, filesize=18.6 K 2023-04-17 12:57:48,247 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/4358d499c58d4eebb6b2136fd3448337 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337 2023-04-17 12:57:48,272 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337, entries=100, sequenceid=316, filesize=9.5 K 2023-04-17 12:57:48,276 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/112b856f99e244369824dce8aeef3f78 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78 2023-04-17 12:57:48,305 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78, entries=100, sequenceid=316, filesize=9.5 K 2023-04-17 12:57:48,308 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=60.38 KB/61830 for 26c16d52fc9da7952b2d73253571802e in 320ms, sequenceid=316, compaction requested=true 2023-04-17 12:57:48,308 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:48,308 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:48,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:48,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:48,309 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:48,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:48,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:48,315 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:48,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fef075c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:48,319 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:48,319 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,320 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:48,320 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:48,320 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:48,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,321 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:48,322 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:48,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:48,325 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:48,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:48,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:48,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:48,325 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:48,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:48,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:48,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:48,344 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:48,344 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:48,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736328388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736328388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,445 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:48,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:48,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:48,470 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:48,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:48,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:48,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,473 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,473 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:48,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:48,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:48,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:48,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:48,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:48,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:48,485 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,485 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:48,485 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:48,485 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:48,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:48,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:48,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:48,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:48,489 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736328491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736328491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:48,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@132cdc26[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:48,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:48,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,495 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:48,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:48,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:48,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:48,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:48,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,497 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:48,497 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:48,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:48,500 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:48,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:48,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:48,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:48,500 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:48,501 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,501 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,516 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,516 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:48,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:48,517 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:48,517 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:48,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:48,617 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:48,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:48,628 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:48,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:48,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:48,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,629 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,630 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,630 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,630 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:48,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:48,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:48,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:48,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:48,641 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:48,641 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:48,641 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,641 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:48,641 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:48,641 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,641 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:48,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:48,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:48,642 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:48,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,643 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,643 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:48,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bfddd03[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:48,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:48,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,645 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:48,645 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,645 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:48,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:48,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:48,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:48,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:48,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:48,650 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:48,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:48,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:48,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:48,650 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:48,652 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,680 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:48,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:48,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,684 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:48,684 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:48,681 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,681 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:48,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736328697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736328702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,784 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:48,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:48,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:48,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:48,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:48,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:48,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:48,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:48,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:48,814 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:48,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:48,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:48,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:48,818 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e7bafec4e4ff45ebab68dd130ccb16be to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e7bafec4e4ff45ebab68dd130ccb16be 2023-04-17 12:57:48,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:48,828 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,828 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:48,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:48,828 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:48,828 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,828 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:48,829 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:48,829 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,829 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:48,829 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:48,830 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:48,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f66e021[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:48,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:48,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,832 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:48,832 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:48,832 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:48,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:48,833 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:48,833 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,833 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:48,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:48,836 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:48,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:48,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:48,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:48,836 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:48,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,837 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,839 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,843 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:48,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:48,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,843 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:48,843 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:48,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:48,944 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:48,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:48,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:48,947 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:48,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:48,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:48,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,948 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,948 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,948 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:48,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:48,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:48,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:48,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:48,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:48,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:48,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:48,951 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:48,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:48,952 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:48,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b4c776[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:48,952 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,953 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:48,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:48,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:48,953 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,953 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:48,954 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,954 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,954 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,954 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:48,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:48,958 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:48,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:48,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:48,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:48,958 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:48,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,971 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:48,993 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:48,993 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:48,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:48,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:48,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:48,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:48,995 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:48,995 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:48,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:48,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:48,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:48,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:48,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:49,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:49,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736329000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:49,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736329024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,095 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:49,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:49,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:49,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:49,099 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:49,099 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,099 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:49,099 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:49,100 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,100 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,101 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:49,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:49,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:49,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:49,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:49,104 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:49,104 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,104 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:49,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:49,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:49,112 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,112 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:49,113 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:49,113 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:49,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:49,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:49,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:49,114 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:49,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f3ae05[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:49,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:49,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,121 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,121 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:49,121 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:49,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:49,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:49,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:49,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:49,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:49,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:49,125 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:49,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:49,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:49,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:49,125 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:49,126 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,128 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,128 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,132 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,133 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,142 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,144 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:49,144 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:49,144 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:49,226 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/9d8f23413da74429924b8f607fdec15b 2023-04-17 12:57:49,246 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:49,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:49,281 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:49,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:49,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:49,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,283 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:49,284 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,287 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,288 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:49,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:49,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:49,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:49,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:49,322 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,322 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:49,322 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:49,322 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:49,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:49,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,324 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:49,324 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e725c1a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:49,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:49,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:49,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:49,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:49,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:49,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:49,330 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,331 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:49,335 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:49,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:49,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:49,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:49,335 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:49,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,337 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,337 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,339 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,347 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:49,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:49,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,348 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:49,348 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,448 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:49,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:49,453 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:49,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:49,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:49,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:49,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:49,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:49,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:49,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:49,468 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:49,468 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:49,472 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,472 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:49,472 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:49,472 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:49,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:49,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:49,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:49,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:49,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645965dd[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:49,479 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:49,479 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,480 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:49,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:49,480 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:49,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:49,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:49,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:49,484 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:49,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:49,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:49,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:49,484 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:49,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,492 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,499 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,499 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,506 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:49,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:49,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,507 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:49,507 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:49,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:49,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736329507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:49,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736329531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,609 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:49,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:49,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:49,618 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:49,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:49,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:49,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:49,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:49,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:49,640 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:49,641 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,641 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:49,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:49,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:49,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:49,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:49,661 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,662 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,662 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3ae3cb[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:49,664 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:49,664 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,664 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:49,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:49,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:49,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:49,675 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:49,675 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,675 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:49,676 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:49,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:49,678 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:49,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:49,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:49,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:49,678 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:49,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:49,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,681 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:49,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:49,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:49,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:49,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:49,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:49,685 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:49,685 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:49,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:49,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/d0c41fa267c9403fa0929b4941566025 2023-04-17 12:57:49,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5b4c67fa9923438d9643f2c4e2fedf31 2023-04-17 12:57:49,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/9d8f23413da74429924b8f607fdec15b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b 2023-04-17 12:57:49,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b, entries=500, sequenceid=329, filesize=73.6 K 2023-04-17 12:57:49,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/d0c41fa267c9403fa0929b4941566025 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025 2023-04-17 12:57:49,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025, entries=150, sequenceid=329, filesize=11.9 K 2023-04-17 12:57:49,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5b4c67fa9923438d9643f2c4e2fedf31 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31 2023-04-17 12:57:49,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31, entries=150, sequenceid=329, filesize=11.9 K 2023-04-17 12:57:49,764 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 26c16d52fc9da7952b2d73253571802e in 1453ms, sequenceid=329, compaction requested=true 2023-04-17 12:57:49,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:49,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:49,765 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:49,765 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:49,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:49,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:49,768 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 198876 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:49,768 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/A is initiating minor compaction (all files) 2023-04-17 12:57:49,768 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/A in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,768 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ff39155b260242dab2bf72f5a51361dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=194.2 K 2023-04-17 12:57:49,769 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ff39155b260242dab2bf72f5a51361dc, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736261033 2023-04-17 12:57:49,770 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 108d88d5a8e0413b921059e5b402af3e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681736263397 2023-04-17 12:57:49,771 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70950 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:49,771 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/B is initiating minor compaction (all files) 2023-04-17 12:57:49,771 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/B in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,771 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/a7bdf258cd004d3aae52335cb60e127f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=69.3 K 2023-04-17 12:57:49,772 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a7bdf258cd004d3aae52335cb60e127f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736261033 2023-04-17 12:57:49,772 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 68f1194b03094a9ea253d5c9a7937be4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736264572 2023-04-17 12:57:49,772 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ef12b7c6f05147d38f2c53a6a0accd53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681736263397 2023-04-17 12:57:49,772 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting fcaa8740896b497e81d6a66b5bf7a1ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681736265835 2023-04-17 12:57:49,773 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ed22039b730c4c6e920f57d679890aff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736264572 2023-04-17 12:57:49,773 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8ea54e5907d34aee98f16a3321acaeda, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681736267042 2023-04-17 12:57:49,773 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ea7e1c6e05c1477aa148d1c4fc15091c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681736265835 2023-04-17 12:57:49,773 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4358d499c58d4eebb6b2136fd3448337, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681736267042 2023-04-17 12:57:49,774 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 9d8f23413da74429924b8f607fdec15b, keycount=500, bloomtype=ROW, size=73.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268204 2023-04-17 12:57:49,774 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d0c41fa267c9403fa0929b4941566025, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268266 2023-04-17 12:57:49,786 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:49,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:49,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:49,793 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#B#compaction#171 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:49,796 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:49,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:49,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:49,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:49,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:49,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:49,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:49,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:49,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:49,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:49,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:49,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:49,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:49,842 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#A#compaction#172 average throughput is 0.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:49,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:49,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:49,844 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,844 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:49,844 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:49,845 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:49,893 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/9fea9a5e69a24579a7825067a02b5736 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9fea9a5e69a24579a7825067a02b5736 2023-04-17 12:57:49,904 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/B of 26c16d52fc9da7952b2d73253571802e into 9fea9a5e69a24579a7825067a02b5736(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:49,904 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:49,904 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/B, priority=10, startTime=1681736269765; duration=0sec 2023-04-17 12:57:49,904 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:49,904 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:57:49,907 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70950 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:57:49,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/C is initiating minor compaction (all files) 2023-04-17 12:57:49,908 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/C in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:49,908 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/4337044f60974dc28be9cfa5f8810fc2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=69.3 K 2023-04-17 12:57:49,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4337044f60974dc28be9cfa5f8810fc2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736261033 2023-04-17 12:57:49,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 47039e06fcbd4042bc3c548192e78917, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681736263397 2023-04-17 12:57:49,910 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5f92fee0e09f4b8f89f69b21f2c29250, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736264572 2023-04-17 12:57:49,911 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 93edb30d06ec40b380094b50f994b0aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681736265835 2023-04-17 12:57:49,915 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 112b856f99e244369824dce8aeef3f78, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681736267042 2023-04-17 12:57:49,916 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5b4c67fa9923438d9643f2c4e2fedf31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268266 2023-04-17 12:57:49,933 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417bea95f750f47489b80ee2f7f87d29988 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bea95f750f47489b80ee2f7f87d29988 2023-04-17 12:57:49,950 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#C#compaction#174 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:50,354 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/63d303be6d00452bb062663e0fa76a31 2023-04-17 12:57:50,434 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/17a641c7318b4c8995b573b62cf779f8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/17a641c7318b4c8995b573b62cf779f8 2023-04-17 12:57:50,443 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/82bf7bf78a8c47a3ac7fc8dd4da9294d 2023-04-17 12:57:50,467 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/C of 26c16d52fc9da7952b2d73253571802e into 17a641c7318b4c8995b573b62cf779f8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:50,467 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:50,467 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/C, priority=10, startTime=1681736269765; duration=0sec 2023-04-17 12:57:50,468 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:50,526 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:50,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:50,539 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/833a840d00a94d8995d359ddede598f3 2023-04-17 12:57:50,548 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/63d303be6d00452bb062663e0fa76a31 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31 2023-04-17 12:57:50,555 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31, entries=150, sequenceid=348, filesize=25.4 K 2023-04-17 12:57:50,557 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/82bf7bf78a8c47a3ac7fc8dd4da9294d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d 2023-04-17 12:57:50,572 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d, entries=150, sequenceid=348, filesize=11.9 K 2023-04-17 12:57:50,580 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/833a840d00a94d8995d359ddede598f3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3 2023-04-17 12:57:50,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736330616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736330622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,631 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3, entries=150, sequenceid=348, filesize=11.9 K 2023-04-17 12:57:50,632 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 26c16d52fc9da7952b2d73253571802e in 788ms, sequenceid=348, compaction requested=false 2023-04-17 12:57:50,632 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:50,632 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:50,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:50,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:50,632 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:50,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:50,636 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,636 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,637 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:50,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74efef[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:50,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:50,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,639 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,639 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:50,644 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:50,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:50,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:50,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:50,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:50,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:50,646 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:50,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:50,654 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:50,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:50,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:50,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:50,655 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:50,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:50,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:50,657 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,659 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:50,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:50,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:50,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,663 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:50,663 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:50,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 12:57:50,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:50,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:50,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:50,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:50,763 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:50,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:50,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:57:50,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:57:50,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:57:50,791 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:57:50,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:50,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:57:50,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:57:50,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:50,793 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/ef9ed0ce276841268947fad853da9808 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ef9ed0ce276841268947fad853da9808 2023-04-17 12:57:50,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:50,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:50,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:57:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:50,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:57:50,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:50,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:57:50,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:57:50,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:57:50,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:57:50,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:57:50,817 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:57:50,823 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:50,823 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. started... 2023-04-17 12:57:50,823 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. as already flushing 2023-04-17 12:57:50,823 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:57:50,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:57:50,824 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/A of 26c16d52fc9da7952b2d73253571802e into ef9ed0ce276841268947fad853da9808(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 12:57:50,824 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:50,825 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/A, priority=10, startTime=1681736269765; duration=1sec 2023-04-17 12:57:50,825 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,825 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:50,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:57:50,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:57:50,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2ae177[Count = 0] remaining members to acquire global barrier 2023-04-17 12:57:50,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:57:50,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,831 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:50,831 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:57:50,831 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:57:50,831 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:57:50,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:57:50,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,836 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:57:50,836 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:50,836 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:57:50,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:57:50,838 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:57:50,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:57:50,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:57:50,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:57:50,838 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:57:50,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:50,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:57:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:57:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:57:50,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736330851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736330854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:57:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:57:50,904 DEBUG [Thread-839] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:62647 2023-04-17 12:57:50,904 DEBUG [Thread-839] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:57:50,915 DEBUG [Thread-837] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486fba63 to 127.0.0.1:62647 2023-04-17 12:57:50,915 DEBUG [Thread-837] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:50,916 DEBUG [Thread-843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52902392 to 127.0.0.1:62647 2023-04-17 12:57:50,917 DEBUG [Thread-843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:50,920 DEBUG [Thread-841] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:62647 2023-04-17 12:57:50,920 DEBUG [Thread-841] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736330957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:50,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736330957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:57:51,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:57:51,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:57:51,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:57:51,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:57:51,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:57:51,146 DEBUG [Thread-836] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:57:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:57:51,146 DEBUG [Thread-836] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:57:51,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,147 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:51,147 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:57:51,147 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:57:51,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:57:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:57:51,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736331159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736331162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,219 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d8eb6a016db24de2925295abe8ebb1f8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d8eb6a016db24de2925295abe8ebb1f8 2023-04-17 12:57:51,247 DEBUG [Thread-836] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:57:51,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:57:51,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736331461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736331463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,626 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/c3f2195ab6fd4df5a594d1853701f0e3 2023-04-17 12:57:51,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33250 deadline: 1681736331966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:51,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:57:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33230 deadline: 1681736331967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:52,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/9ece94e74e8b4a8eb187d4f8e3dca6e2 2023-04-17 12:57:52,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/9b8df5d0b1904a1fabbf976b05c48904 2023-04-17 12:57:52,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/c3f2195ab6fd4df5a594d1853701f0e3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3 2023-04-17 12:57:52,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3, entries=200, sequenceid=364, filesize=32.3 K 2023-04-17 12:57:52,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/9ece94e74e8b4a8eb187d4f8e3dca6e2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2 2023-04-17 12:57:52,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2, entries=150, sequenceid=364, filesize=11.9 K 2023-04-17 12:57:52,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/9b8df5d0b1904a1fabbf976b05c48904 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904 2023-04-17 12:57:52,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904, entries=150, sequenceid=364, filesize=11.9 K 2023-04-17 12:57:52,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 26c16d52fc9da7952b2d73253571802e in 1740ms, sequenceid=364, compaction requested=true 2023-04-17 12:57:52,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:52,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:52,473 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:52,473 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:52,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:52,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:52,474 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:52,474 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:52,474 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/B is initiating minor compaction (all files) 2023-04-17 12:57:52,475 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/A is initiating minor compaction (all files) 2023-04-17 12:57:52,475 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/B in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:52,475 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/A in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:52,475 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ef9ed0ce276841268947fad853da9808, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=83.8 K 2023-04-17 12:57:52,475 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9fea9a5e69a24579a7825067a02b5736, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=36.4 K 2023-04-17 12:57:52,475 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ef9ed0ce276841268947fad853da9808, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268266 2023-04-17 12:57:52,475 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9fea9a5e69a24579a7825067a02b5736, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268266 2023-04-17 12:57:52,475 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 63d303be6d00452bb062663e0fa76a31, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681736268375 2023-04-17 12:57:52,475 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 82bf7bf78a8c47a3ac7fc8dd4da9294d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681736268375 2023-04-17 12:57:52,476 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c3f2195ab6fd4df5a594d1853701f0e3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681736270556 2023-04-17 12:57:52,476 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9ece94e74e8b4a8eb187d4f8e3dca6e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681736270556 2023-04-17 12:57:52,482 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#B#compaction#181 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:52,486 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#A#compaction#180 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:52,493 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/d40205885a46477a8d6918ccb88bb938 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d40205885a46477a8d6918ccb88bb938 2023-04-17 12:57:52,498 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/B of 26c16d52fc9da7952b2d73253571802e into d40205885a46477a8d6918ccb88bb938(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:52,498 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:52,498 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/B, priority=13, startTime=1681736272473; duration=0sec 2023-04-17 12:57:52,499 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:57:52,499 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:57:52,500 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:57:52,500 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 26c16d52fc9da7952b2d73253571802e/C is initiating minor compaction (all files) 2023-04-17 12:57:52,500 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 26c16d52fc9da7952b2d73253571802e/C in TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:52,500 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/17a641c7318b4c8995b573b62cf779f8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp, totalSize=36.4 K 2023-04-17 12:57:52,501 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 17a641c7318b4c8995b573b62cf779f8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681736268266 2023-04-17 12:57:52,501 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 833a840d00a94d8995d359ddede598f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681736268375 2023-04-17 12:57:52,502 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9b8df5d0b1904a1fabbf976b05c48904, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681736270556 2023-04-17 12:57:52,508 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 26c16d52fc9da7952b2d73253571802e#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:57:52,518 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/21bd00c5e2144e928988e087e86e717f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/21bd00c5e2144e928988e087e86e717f 2023-04-17 12:57:52,524 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/C of 26c16d52fc9da7952b2d73253571802e into 21bd00c5e2144e928988e087e86e717f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:57:52,524 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:52,524 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/C, priority=13, startTime=1681736272473; duration=0sec 2023-04-17 12:57:52,524 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:52,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:52,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:57:52,977 DEBUG [Thread-826] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e3bbbd0 to 127.0.0.1:62647 2023-04-17 12:57:52,977 DEBUG [Thread-832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7acc1387 to 127.0.0.1:62647 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:52,978 DEBUG [Thread-832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:52,978 DEBUG [Thread-826] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:52,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:53,392 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c9609c17844941b9962251b16e848dcd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9609c17844941b9962251b16e848dcd 2023-04-17 12:57:53,711 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/83351a0189d84032ba56587bf6ee1770 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/83351a0189d84032ba56587bf6ee1770 2023-04-17 12:57:53,716 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 26c16d52fc9da7952b2d73253571802e/A of 26c16d52fc9da7952b2d73253571802e into 83351a0189d84032ba56587bf6ee1770(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 12:57:53,717 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:53,717 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e., storeName=26c16d52fc9da7952b2d73253571802e/A, priority=13, startTime=1681736272473; duration=1sec 2023-04-17 12:57:53,717 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:57:53,814 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/5333ad8abce04076b2faa70c97438c93 2023-04-17 12:57:54,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/30b9edc0ea114266a42a0095eea754d3 2023-04-17 12:57:54,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/034be401013f49408365dd68b487e220 2023-04-17 12:57:54,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/5333ad8abce04076b2faa70c97438c93 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/5333ad8abce04076b2faa70c97438c93 2023-04-17 12:57:54,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/5333ad8abce04076b2faa70c97438c93, entries=150, sequenceid=387, filesize=25.4 K 2023-04-17 12:57:54,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/30b9edc0ea114266a42a0095eea754d3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/30b9edc0ea114266a42a0095eea754d3 2023-04-17 12:57:54,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/30b9edc0ea114266a42a0095eea754d3, entries=150, sequenceid=387, filesize=11.9 K 2023-04-17 12:57:54,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/034be401013f49408365dd68b487e220 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/034be401013f49408365dd68b487e220 2023-04-17 12:57:54,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/034be401013f49408365dd68b487e220, entries=150, sequenceid=387, filesize=11.9 K 2023-04-17 12:57:54,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 26c16d52fc9da7952b2d73253571802e in 1305ms, sequenceid=387, compaction requested=false 2023-04-17 12:57:54,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:54,368 DEBUG [Thread-834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7be739cc to 127.0.0.1:62647 2023-04-17 12:57:54,368 DEBUG [Thread-834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:56,933 DEBUG [regionserver/jenkins-hbase17:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-17 12:57:57,744 DEBUG [Thread-830] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b3311fc to 127.0.0.1:62647 2023-04-17 12:57:57,744 DEBUG [Thread-830] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:57,749 DEBUG [Thread-828] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d781def to 127.0.0.1:62647 2023-04-17 12:57:57,749 DEBUG [Thread-828] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:57,749 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:57:57,749 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-17 12:57:57,749 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-04-17 12:57:57,749 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2766 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2750 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1209 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3626 rows 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1227 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3674 rows 2023-04-17 12:57:57,750 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:57:57,751 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7306f12a to 127.0.0.1:62647 2023-04-17 12:57:57,751 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:57:57,753 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:57:57,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:57:57,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:57,757 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736277757"}]},"ts":"1681736277757"} 2023-04-17 12:57:57,759 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:57:57,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:57,937 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:57:57,938 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, UNASSIGN}] 2023-04-17 12:57:57,941 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, UNASSIGN 2023-04-17 12:57:57,942 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:57:57,942 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736277942"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736277942"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736277942"}]},"ts":"1681736277942"} 2023-04-17 12:57:57,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:57:58,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:58,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:58,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 26c16d52fc9da7952b2d73253571802e, disabling compactions & flushes 2023-04-17 12:57:58,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:58,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:58,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. after waiting 0 ms 2023-04-17 12:57:58,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:58,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 26c16d52fc9da7952b2d73253571802e 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 12:57:58,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=A 2023-04-17 12:57:58,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:58,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=B 2023-04-17 12:57:58,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:58,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 26c16d52fc9da7952b2d73253571802e, store=C 2023-04-17 12:57:58,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:57:58,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:58,523 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178a7783198add4ed3b0f3863a8c35a434 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178a7783198add4ed3b0f3863a8c35a434 2023-04-17 12:57:58,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=394, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/a2ebb50c6d224f77a35b2f0e964309d5 2023-04-17 12:57:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:58,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/8cf57944285844fa919953eee6b0b767 2023-04-17 12:57:59,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5418028b376f4c72802b1df4286a108a 2023-04-17 12:57:59,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/A/a2ebb50c6d224f77a35b2f0e964309d5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/a2ebb50c6d224f77a35b2f0e964309d5 2023-04-17 12:57:59,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/a2ebb50c6d224f77a35b2f0e964309d5, entries=50, sequenceid=394, filesize=11.7 K 2023-04-17 12:57:59,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/B/8cf57944285844fa919953eee6b0b767 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/8cf57944285844fa919953eee6b0b767 2023-04-17 12:57:59,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/8cf57944285844fa919953eee6b0b767, entries=50, sequenceid=394, filesize=7.1 K 2023-04-17 12:57:59,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/.tmp/C/5418028b376f4c72802b1df4286a108a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5418028b376f4c72802b1df4286a108a 2023-04-17 12:57:59,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5418028b376f4c72802b1df4286a108a, entries=50, sequenceid=394, filesize=7.1 K 2023-04-17 12:57:59,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 26c16d52fc9da7952b2d73253571802e in 1288ms, sequenceid=394, compaction requested=true 2023-04-17 12:57:59,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/cd3b1bf625f942d3a5f24adf1b8fa2dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/4d6113645fa348f0800a12e73779cb40, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ff39155b260242dab2bf72f5a51361dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ef9ed0ce276841268947fad853da9808, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3] to archive 2023-04-17 12:57:59,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:59,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/3b4d2e49b05348e0acc4c2d249fc02f1 2023-04-17 12:57:59,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/e783060c6eee4e93995ebe0d40d065cb 2023-04-17 12:57:59,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/db67b3e01a0a4b99b0812267fb095e0b 2023-04-17 12:57:59,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/94e8c50d44684873a6dd12890fb7db6e 2023-04-17 12:57:59,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/cd3b1bf625f942d3a5f24adf1b8fa2dc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/cd3b1bf625f942d3a5f24adf1b8fa2dc 2023-04-17 12:57:59,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/384e29545a084ec2b42e4adb75f7fd71 2023-04-17 12:57:59,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/882e0dea612746b3b8b47f54276cfcf8 2023-04-17 12:57:59,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/61d126e6abca457faa68d3ef47280814 2023-04-17 12:57:59,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/16f33a9e49ce4a949e8fd048f6a34d73 2023-04-17 12:57:59,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9f4ff4378c854566973aafe55de5d0a3 2023-04-17 12:57:59,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/4d6113645fa348f0800a12e73779cb40 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/4d6113645fa348f0800a12e73779cb40 2023-04-17 12:57:59,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c5fdd11b66ad481c95bd6d9058c6fd3f 2023-04-17 12:57:59,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/7f00ff30c01c4b3cb10aba1785de165b 2023-04-17 12:57:59,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/d8b5c942e4ce4c9691582150ce06e557 2023-04-17 12:57:59,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ff39155b260242dab2bf72f5a51361dc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ff39155b260242dab2bf72f5a51361dc 2023-04-17 12:57:59,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/108d88d5a8e0413b921059e5b402af3e 2023-04-17 12:57:59,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/68f1194b03094a9ea253d5c9a7937be4 2023-04-17 12:57:59,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/fcaa8740896b497e81d6a66b5bf7a1ed 2023-04-17 12:57:59,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/8ea54e5907d34aee98f16a3321acaeda 2023-04-17 12:57:59,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/9d8f23413da74429924b8f607fdec15b 2023-04-17 12:57:59,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ef9ed0ce276841268947fad853da9808 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/ef9ed0ce276841268947fad853da9808 2023-04-17 12:57:59,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/63d303be6d00452bb062663e0fa76a31 2023-04-17 12:57:59,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/c3f2195ab6fd4df5a594d1853701f0e3 2023-04-17 12:57:59,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/7d995db14144477da65e2d0d06b2db2c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/e28a71d230cd453f9d5e67406604df7d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/a7bdf258cd004d3aae52335cb60e127f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9fea9a5e69a24579a7825067a02b5736, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2] to archive 2023-04-17 12:57:59,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:59,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/794579ebb960472a994d9a6adbb8410b 2023-04-17 12:57:59,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c1e542ddffc9408e8a3412c104a343ff 2023-04-17 12:57:59,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/b36d149810dd47f6b2ec77aa985a45c4 2023-04-17 12:57:59,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/7d995db14144477da65e2d0d06b2db2c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/7d995db14144477da65e2d0d06b2db2c 2023-04-17 12:57:59,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/09d64be2fd494eca853e005678cecdfe 2023-04-17 12:57:59,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/fc677b24298e46e092c2d7be2be84caf 2023-04-17 12:57:59,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/c5d15d666bef4b75bfd691ada429d886 2023-04-17 12:57:59,446 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3eb15cd11f9a4a5abd9b41b5bc4ff25a 2023-04-17 12:57:59,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/87f94af240514e64a861334c741ac321 2023-04-17 12:57:59,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/e28a71d230cd453f9d5e67406604df7d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/e28a71d230cd453f9d5e67406604df7d 2023-04-17 12:57:59,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/164a595b3cef49f59ec4f23d18aa4640 2023-04-17 12:57:59,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/3ab409f0bbb34de7aabf421beed01976 2023-04-17 12:57:59,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1b9e0c0d507344b5b0c65687a22e4999 2023-04-17 12:57:59,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/a7bdf258cd004d3aae52335cb60e127f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/a7bdf258cd004d3aae52335cb60e127f 2023-04-17 12:57:59,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/1f5517275d9c45bd86660c1100035f85 2023-04-17 12:57:59,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ef12b7c6f05147d38f2c53a6a0accd53 2023-04-17 12:57:59,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ed22039b730c4c6e920f57d679890aff 2023-04-17 12:57:59,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/ea7e1c6e05c1477aa148d1c4fc15091c 2023-04-17 12:57:59,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/4358d499c58d4eebb6b2136fd3448337 2023-04-17 12:57:59,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9fea9a5e69a24579a7825067a02b5736 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9fea9a5e69a24579a7825067a02b5736 2023-04-17 12:57:59,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d0c41fa267c9403fa0929b4941566025 2023-04-17 12:57:59,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/82bf7bf78a8c47a3ac7fc8dd4da9294d 2023-04-17 12:57:59,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/9ece94e74e8b4a8eb187d4f8e3dca6e2 2023-04-17 12:57:59,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/29790280e8b34692aa54f9595a43bdf5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/e44f80dae1fd4aee8f4c0f4aca7f049a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/4337044f60974dc28be9cfa5f8810fc2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/17a641c7318b4c8995b573b62cf779f8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904] to archive 2023-04-17 12:57:59,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:57:59,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1de881f987804cb99ccc746787570534 2023-04-17 12:57:59,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/d25d2fa415ed4c41b150c07b42b33cd7 2023-04-17 12:57:59,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/a7d1f95857b845f7804140ae55597c28 2023-04-17 12:57:59,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/29790280e8b34692aa54f9595a43bdf5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/29790280e8b34692aa54f9595a43bdf5 2023-04-17 12:57:59,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c69bdf8f8f264a75bba00e7d52f32b84 2023-04-17 12:57:59,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/1458830ca1764432b54c6670c054ad48 2023-04-17 12:57:59,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/f2b7c588fd3e45459f7fd7175034bc25 2023-04-17 12:57:59,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/c8708179e88544baa161f34886f97223 2023-04-17 12:57:59,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/bf4d97410151490f9554d840e03477cc 2023-04-17 12:57:59,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/e44f80dae1fd4aee8f4c0f4aca7f049a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/e44f80dae1fd4aee8f4c0f4aca7f049a 2023-04-17 12:57:59,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/23c81af2806d4417904fa200746107c9 2023-04-17 12:57:59,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/ef988e3a57f14f9e847ed4e13252934e 2023-04-17 12:57:59,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/97a41f60da4e4d6a8bd68dfa870a7b0a 2023-04-17 12:57:59,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/4337044f60974dc28be9cfa5f8810fc2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/4337044f60974dc28be9cfa5f8810fc2 2023-04-17 12:57:59,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/8378a82a4a3f4eb1ac47d58d9e1fc7f4 2023-04-17 12:57:59,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/47039e06fcbd4042bc3c548192e78917 2023-04-17 12:57:59,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5f92fee0e09f4b8f89f69b21f2c29250 2023-04-17 12:57:59,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/93edb30d06ec40b380094b50f994b0aa 2023-04-17 12:57:59,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/112b856f99e244369824dce8aeef3f78 2023-04-17 12:57:59,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/17a641c7318b4c8995b573b62cf779f8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/17a641c7318b4c8995b573b62cf779f8 2023-04-17 12:57:59,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5b4c67fa9923438d9643f2c4e2fedf31 2023-04-17 12:57:59,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/833a840d00a94d8995d359ddede598f3 2023-04-17 12:57:59,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/9b8df5d0b1904a1fabbf976b05c48904 2023-04-17 12:57:59,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits/397.seqid, newMaxSeqId=397, maxSeqId=4 2023-04-17 12:57:59,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e. 2023-04-17 12:57:59,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 26c16d52fc9da7952b2d73253571802e: 2023-04-17 12:57:59,512 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:59,512 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=26c16d52fc9da7952b2d73253571802e, regionState=CLOSED 2023-04-17 12:57:59,512 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736279512"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736279512"}]},"ts":"1681736279512"} 2023-04-17 12:57:59,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-17 12:57:59,515 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 26c16d52fc9da7952b2d73253571802e, server=jenkins-hbase17.apache.org,40215,1681736213280 in 1.5700 sec 2023-04-17 12:57:59,517 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-17 12:57:59,517 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=26c16d52fc9da7952b2d73253571802e, UNASSIGN in 1.5770 sec 2023-04-17 12:57:59,518 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736279517"}]},"ts":"1681736279517"} 2023-04-17 12:57:59,518 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:57:59,519 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:57:59,521 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7670 sec 2023-04-17 12:57:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 12:57:59,863 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-17 12:57:59,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:57:59,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,867 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 12:57:59,867 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,873 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:59,876 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits] 2023-04-17 12:57:59,878 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/5333ad8abce04076b2faa70c97438c93 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/5333ad8abce04076b2faa70c97438c93 2023-04-17 12:57:59,880 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/83351a0189d84032ba56587bf6ee1770 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/83351a0189d84032ba56587bf6ee1770 2023-04-17 12:57:59,881 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/a2ebb50c6d224f77a35b2f0e964309d5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/A/a2ebb50c6d224f77a35b2f0e964309d5 2023-04-17 12:57:59,885 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/30b9edc0ea114266a42a0095eea754d3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/30b9edc0ea114266a42a0095eea754d3 2023-04-17 12:57:59,891 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/8cf57944285844fa919953eee6b0b767 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/8cf57944285844fa919953eee6b0b767 2023-04-17 12:57:59,892 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d40205885a46477a8d6918ccb88bb938 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/B/d40205885a46477a8d6918ccb88bb938 2023-04-17 12:57:59,894 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/034be401013f49408365dd68b487e220 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/034be401013f49408365dd68b487e220 2023-04-17 12:57:59,895 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/21bd00c5e2144e928988e087e86e717f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/21bd00c5e2144e928988e087e86e717f 2023-04-17 12:57:59,896 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5418028b376f4c72802b1df4286a108a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/C/5418028b376f4c72802b1df4286a108a 2023-04-17 12:57:59,900 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits/397.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e/recovered.edits/397.seqid 2023-04-17 12:57:59,901 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/26c16d52fc9da7952b2d73253571802e 2023-04-17 12:57:59,901 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:57:59,902 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:57:59,907 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 12:57:59,911 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170c00a347901a44ed82cbbec84bf964c0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170c00a347901a44ed82cbbec84bf964c0 2023-04-17 12:57:59,912 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171dbbc6e94c1f4249987e04a3ffe42529 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171dbbc6e94c1f4249987e04a3ffe42529 2023-04-17 12:57:59,915 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173021bff5381f489cad3584883ccc5230 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173021bff5381f489cad3584883ccc5230 2023-04-17 12:57:59,916 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417406826f295074c078ff13723f615608c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417406826f295074c078ff13723f615608c 2023-04-17 12:57:59,921 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417466b6d9154e645f685b81aaa09072333 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417466b6d9154e645f685b81aaa09072333 2023-04-17 12:57:59,922 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417486391187fd741078df36d8a0ff1421c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417486391187fd741078df36d8a0ff1421c 2023-04-17 12:57:59,924 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178a7783198add4ed3b0f3863a8c35a434 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178a7783198add4ed3b0f3863a8c35a434 2023-04-17 12:57:59,925 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178c7154eed1d442a7a5fdeb80c2294755 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178c7154eed1d442a7a5fdeb80c2294755 2023-04-17 12:57:59,926 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179dded1c3b4694aedbca5964396a70427 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179dded1c3b4694aedbca5964396a70427 2023-04-17 12:57:59,927 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bea95f750f47489b80ee2f7f87d29988 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bea95f750f47489b80ee2f7f87d29988 2023-04-17 12:57:59,928 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c0668012bbb6416fa0aa8c3a499394a7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c0668012bbb6416fa0aa8c3a499394a7 2023-04-17 12:57:59,929 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c29651ac6ddd40d58f623acbf7e47584 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c29651ac6ddd40d58f623acbf7e47584 2023-04-17 12:57:59,933 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9609c17844941b9962251b16e848dcd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c9609c17844941b9962251b16e848dcd 2023-04-17 12:57:59,934 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c961b27ede4941ab9c2a9c3981559e3e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c961b27ede4941ab9c2a9c3981559e3e 2023-04-17 12:57:59,937 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d8eb6a016db24de2925295abe8ebb1f8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d8eb6a016db24de2925295abe8ebb1f8 2023-04-17 12:57:59,939 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dd4f81588bd942219bdf03852fb180bc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dd4f81588bd942219bdf03852fb180bc 2023-04-17 12:57:59,941 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e22e12a9483149a4a05131168f139cb1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e22e12a9483149a4a05131168f139cb1 2023-04-17 12:57:59,942 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e7bafec4e4ff45ebab68dd130ccb16be to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e7bafec4e4ff45ebab68dd130ccb16be 2023-04-17 12:57:59,943 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eeb98b15c46b4cd38b41b8361fa51575 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eeb98b15c46b4cd38b41b8361fa51575 2023-04-17 12:57:59,944 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f60ae401908b49f094000e883e7c90c1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f60ae401908b49f094000e883e7c90c1 2023-04-17 12:57:59,946 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f89d38969b2f4d81af7613e3191c5ada to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f89d38969b2f4d81af7613e3191c5ada 2023-04-17 12:57:59,946 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:57:59,948 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,951 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:57:59,953 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:57:59,955 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,955 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:57:59,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736279955"}]},"ts":"9223372036854775807"} 2023-04-17 12:57:59,957 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:57:59,957 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 26c16d52fc9da7952b2d73253571802e, NAME => 'TestAcidGuarantees,,1681736247594.26c16d52fc9da7952b2d73253571802e.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:57:59,957 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:57:59,957 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736279957"}]},"ts":"9223372036854775807"} 2023-04-17 12:57:59,962 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:57:59,964 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:57:59,965 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 100 msec 2023-04-17 12:57:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 12:57:59,968 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-17 12:57:59,988 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=226 (was 234), OpenFileDescriptor=540 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=674 (was 573) - SystemLoadAverage LEAK? -, ProcessCount=195 (was 192) - ProcessCount LEAK? -, AvailableMemoryMB=3458 (was 3342) - AvailableMemoryMB LEAK? - 2023-04-17 12:58:00,005 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226, OpenFileDescriptor=540, MaxFileDescriptor=60000, SystemLoadAverage=674, ProcessCount=195, AvailableMemoryMB=3458 2023-04-17 12:58:00,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:58:00,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:58:00,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:00,011 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:58:00,012 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:58:00,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-17 12:58:00,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 12:58:00,015 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,026 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 empty. 2023-04-17 12:58:00,027 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,028 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:58:00,045 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:58:00,047 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2a5e7440639effb3958a1a1b6f36fd53, NAME => 'TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:58:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 12:58:00,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2a5e7440639effb3958a1a1b6f36fd53, disabling compactions & flushes 2023-04-17 12:58:00,465 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. after waiting 0 ms 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,465 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,465 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:00,470 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:58:00,471 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736280471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736280471"}]},"ts":"1681736280471"} 2023-04-17 12:58:00,472 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:58:00,474 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:58:00,475 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736280474"}]},"ts":"1681736280474"} 2023-04-17 12:58:00,476 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:58:00,477 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, ASSIGN}] 2023-04-17 12:58:00,479 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, ASSIGN 2023-04-17 12:58:00,480 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:58:00,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 12:58:00,631 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2a5e7440639effb3958a1a1b6f36fd53, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:00,631 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736280631"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736280631"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736280631"}]},"ts":"1681736280631"} 2023-04-17 12:58:00,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:00,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2a5e7440639effb3958a1a1b6f36fd53, NAME => 'TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:58:00,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:00,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,794 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,797 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A 2023-04-17 12:58:00,798 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A 2023-04-17 12:58:00,798 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:00,798 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a5e7440639effb3958a1a1b6f36fd53 columnFamilyName A 2023-04-17 12:58:00,799 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(310): Store=2a5e7440639effb3958a1a1b6f36fd53/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:00,799 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,800 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B 2023-04-17 12:58:00,800 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B 2023-04-17 12:58:00,801 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:00,801 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a5e7440639effb3958a1a1b6f36fd53 columnFamilyName B 2023-04-17 12:58:00,802 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(310): Store=2a5e7440639effb3958a1a1b6f36fd53/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:00,802 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,803 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C 2023-04-17 12:58:00,803 DEBUG [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C 2023-04-17 12:58:00,803 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:00,803 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a5e7440639effb3958a1a1b6f36fd53 columnFamilyName C 2023-04-17 12:58:00,804 INFO [StoreOpener-2a5e7440639effb3958a1a1b6f36fd53-1] regionserver.HStore(310): Store=2a5e7440639effb3958a1a1b6f36fd53/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:00,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:58:00,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:00,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:58:00,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 2a5e7440639effb3958a1a1b6f36fd53; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68924177, jitterRate=0.02705027163028717}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:58:00,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:00,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., pid=30, masterSystemTime=1681736280783 2023-04-17 12:58:00,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:00,817 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2a5e7440639effb3958a1a1b6f36fd53, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:00,818 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736280817"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736280817"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736280817"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736280817"}]},"ts":"1681736280817"} 2023-04-17 12:58:00,820 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-17 12:58:00,820 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 in 187 msec 2023-04-17 12:58:00,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-17 12:58:00,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, ASSIGN in 343 msec 2023-04-17 12:58:00,823 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:58:00,823 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736280823"}]},"ts":"1681736280823"} 2023-04-17 12:58:00,824 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:58:00,826 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:58:00,828 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 818 msec 2023-04-17 12:58:01,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 12:58:01,122 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-17 12:58:01,122 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x64e0df67 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,128 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30cdd851, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,130 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,131 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:60984, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,138 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:58:01,142 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:58:01,143 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7117e4df to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,156 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136e067d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,157 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,163 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,163 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,175 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,175 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,181 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,181 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5a8a575d to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,193 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e3efec6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,194 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,227 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,228 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,240 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,241 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,258 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,258 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,265 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,266 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5d22a55e to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:01,274 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@993cc7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:01,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:01,293 DEBUG [hconnection-0x4aae601a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,294 DEBUG [hconnection-0x59fa15af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,299 DEBUG [hconnection-0x446974a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,300 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,305 DEBUG [hconnection-0x526b3379-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,305 DEBUG [hconnection-0x4fc393eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,311 DEBUG [hconnection-0x3eac37b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,312 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,318 DEBUG [hconnection-0x48cbcb49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,322 DEBUG [hconnection-0x6779f45a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,324 DEBUG [hconnection-0x69ab9f6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,335 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:60996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,336 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,336 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,337 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:60998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,337 DEBUG [hconnection-0x49fe1683-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:01,337 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,337 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,337 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:01,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:01,343 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:01,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:01,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:01,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,349 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,349 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:32834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:01,349 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:01,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:01,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:01,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:01,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:01,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:01,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:01,362 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:01,362 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:01,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:01,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:01,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:01,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:58:01,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:01,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:01,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:01,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:01,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:01,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:01,416 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:01,416 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:01,416 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:01,416 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:01,416 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736341421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736341425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736341426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,436 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:01,437 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:01,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736341427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736341434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,446 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,447 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,447 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:01,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637212a2[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:01,448 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:01,449 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:01,454 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:01,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:01,455 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:01,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:01,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:01,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:01,456 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:01,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:01,458 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:01,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:01,459 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:01,459 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:01,459 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:01,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,460 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,462 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,462 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:01,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,464 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:01,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:01,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:01,466 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:01,466 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:01,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:01,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:01,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:01,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/a1ab21b4f2354fe0a81b551c6c7ace1a 2023-04-17 12:58:01,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736341534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736341534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736341538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736341541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736341547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,570 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:01,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:01,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:01,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/291fb4fa435c4e1f800a224379858e7b 2023-04-17 12:58:01,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:01,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:01,635 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:01,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:01,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:01,636 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,636 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:01,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:01,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:01,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:01,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:01,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:01,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:01,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:01,651 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:01,651 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:01,651 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:01,651 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:01,651 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:01,655 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:01,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cc81bc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:01,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:01,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:01,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:01,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:01,663 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,663 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:01,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:01,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:01,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:01,665 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:01,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:01,668 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:01,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:01,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:01,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:01,668 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:01,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,671 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,671 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:01,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:01,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:01,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,683 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:01,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:01,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:01,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:01,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,684 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:01,684 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:01,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:01,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:01,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/ac2e6f3f15b346008184a2079405781f 2023-04-17 12:58:01,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/a1ab21b4f2354fe0a81b551c6c7ace1a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a 2023-04-17 12:58:01,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736341742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736341745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736341746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736341754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:01,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736341754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 12:58:01,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/291fb4fa435c4e1f800a224379858e7b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b 2023-04-17 12:58:01,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 12:58:01,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/ac2e6f3f15b346008184a2079405781f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f 2023-04-17 12:58:01,784 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:01,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:01,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:01,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 12:58:01,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2a5e7440639effb3958a1a1b6f36fd53 in 415ms, sequenceid=15, compaction requested=false 2023-04-17 12:58:01,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:01,804 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:01,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:01,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:01,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,810 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:01,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:01,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:01,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:01,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:01,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:01,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:01,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:01,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:01,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:01,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:01,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:01,833 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:01,833 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:01,833 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:01,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:02,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:02,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:02,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736342099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736342100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736342103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736342105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736342107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736342214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736342214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736342219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736342219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736342222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,289 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/5151f6107e0b4972912e6115fa5950d8 2023-04-17 12:58:02,358 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d7465319c2094be6bfec92834cc80fba 2023-04-17 12:58:02,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736342422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736342424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736342427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736342426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736342438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736342730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736342731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736342735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736342739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:02,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736342760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,838 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8fe7072b43b04fdbbcbce239e657c78f 2023-04-17 12:58:02,853 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/5151f6107e0b4972912e6115fa5950d8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8 2023-04-17 12:58:02,864 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 12:58:02,865 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d7465319c2094be6bfec92834cc80fba as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba 2023-04-17 12:58:02,877 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 12:58:02,878 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8fe7072b43b04fdbbcbce239e657c78f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f 2023-04-17 12:58:02,882 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 12:58:02,883 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 2a5e7440639effb3958a1a1b6f36fd53 in 1050ms, sequenceid=33, compaction requested=false 2023-04-17 12:58:02,883 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:02,883 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:02,883 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:02,883 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:02,883 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,884 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:02,884 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:02,885 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,885 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:02,885 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,885 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:02,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:02,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee002fd[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:02,887 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:02,887 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,888 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,888 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:02,888 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:02,888 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:02,889 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,889 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:02,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:02,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:02,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:02,890 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:02,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:02,893 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:02,893 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:02,893 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:02,893 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:02,894 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:02,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:02,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,896 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:02,897 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,901 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:02,901 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:02,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:02,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:02,901 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:02,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:02,901 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:02,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:02,902 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:02,902 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:03,004 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:03,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:03,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:03,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:03,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:03,014 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:03,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:03,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:03,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,018 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,019 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,019 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:03,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:03,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:03,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:03,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:03,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:03,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:03,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:03,022 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,022 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:03,023 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 12:58:03,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:03,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:03,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:03,024 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,057 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/c7b3aa18e48a4be580c6ffade05a5dfa 2023-04-17 12:58:03,097 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/148e1044bb004d72abdf32b52b9fd078 2023-04-17 12:58:03,126 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/35b9545b1a254c60a8f707db8f157318 2023-04-17 12:58:03,152 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/c7b3aa18e48a4be580c6ffade05a5dfa as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa 2023-04-17 12:58:03,159 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 12:58:03,161 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/148e1044bb004d72abdf32b52b9fd078 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078 2023-04-17 12:58:03,170 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 12:58:03,178 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/35b9545b1a254c60a8f707db8f157318 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318 2023-04-17 12:58:03,190 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318, entries=150, sequenceid=48, filesize=11.6 K 2023-04-17 12:58:03,195 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 2a5e7440639effb3958a1a1b6f36fd53 in 172ms, sequenceid=48, compaction requested=true 2023-04-17 12:58:03,195 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:03,195 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:03,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:03,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:03,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:03,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:03,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:03,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45da610f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:03,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:03,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:03,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:03,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:03,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:03,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:03,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:03,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:03,204 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:03,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:03,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:03,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:03,205 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:03,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,211 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:03,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:03,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,222 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:03,222 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:03,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:58:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:03,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:03,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:03,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/3bfa036b61ce4b51ab9f597f7a766461 2023-04-17 12:58:03,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736283269, max timestamp=1681736283290Num uniques 859; 2023-04-17 12:58:03,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736283269, max timestamp=1681736283290Num uniques 859; 2023-04-17 12:58:03,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:58:03,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:58:03,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736283280, max timestamp=1681736283290Num uniques 859; 2023-04-17 12:58:03,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:58:03,323 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:03,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:03,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:03,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736343327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736343333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736343333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736343333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736343334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:03,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:03,350 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:03,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:03,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:03,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,352 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:03,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:03,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:03,387 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:03,387 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,387 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:03,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:03,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:03,400 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:03,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:03,402 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,402 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:03,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ce9c14[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:03,413 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:03,413 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,415 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,415 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:03,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:03,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:03,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:03,430 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,440 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:03,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736343439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736343440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736343446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:03,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:03,451 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:03,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:03,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:03,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:03,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:03,453 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,453 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736343446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,453 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,459 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:03,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:03,460 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,460 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:03,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:03,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736343449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,460 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:03,460 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:03,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,561 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:03,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:03,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:03,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:03,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:03,572 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:03,572 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,572 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:03,572 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:03,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,573 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:03,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:03,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:03,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:03,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:03,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:03,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:03,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:03,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:03,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:03,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:03,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:03,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:03,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:03,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:03,601 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:03,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b923a10[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:03,632 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:03,632 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:03,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:03,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:03,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:03,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,635 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:03,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,643 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:03,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:03,644 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:03,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:03,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:03,644 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:03,644 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:03,645 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,645 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,645 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,646 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,656 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:03,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:03,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,657 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,657 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:03,657 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:03,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736343650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736343656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736343656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736343666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:03,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736343666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/27ad2d44db3c42e3ad4b3e2e3f724515 2023-04-17 12:58:03,757 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:03,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:03,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:03,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:03,766 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:03,766 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,766 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:03,766 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:03,767 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,767 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/92700b65fc23400196992352d8a8e21e 2023-04-17 12:58:03,769 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:03,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:03,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/3bfa036b61ce4b51ab9f597f7a766461 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461 2023-04-17 12:58:03,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:03,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:03,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:03,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:03,787 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:03,787 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:03,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:03,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:03,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,788 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:03,788 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:03,788 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,788 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:03,788 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:03,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461, entries=200, sequenceid=59, filesize=13.9 K 2023-04-17 12:58:03,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/27ad2d44db3c42e3ad4b3e2e3f724515 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515 2023-04-17 12:58:03,792 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:03,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515, entries=150, sequenceid=59, filesize=11.6 K 2023-04-17 12:58:03,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/92700b65fc23400196992352d8a8e21e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e 2023-04-17 12:58:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:03,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75bb3275[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:03,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:03,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e, entries=150, sequenceid=59, filesize=11.6 K 2023-04-17 12:58:03,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:03,818 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:03,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:03,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:03,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:03,819 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:03,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:03,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:03,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:03,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:03,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:03,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:03,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127835 for 2a5e7440639effb3958a1a1b6f36fd53 in 573ms, sequenceid=59, compaction requested=true 2023-04-17 12:58:03,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:03,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:03,822 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:03,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:03,823 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:03,823 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:03,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,824 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:03,824 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:03,824 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,824 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=46.4 K 2023-04-17 12:58:03,825 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,825 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:03,825 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,825 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=48.7 K 2023-04-17 12:58:03,825 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 291fb4fa435c4e1f800a224379858e7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681736281362 2023-04-17 12:58:03,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,826 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a1ab21b4f2354fe0a81b551c6c7ace1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681736281362 2023-04-17 12:58:03,826 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 5151f6107e0b4972912e6115fa5950d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681736281418 2023-04-17 12:58:03,826 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d7465319c2094be6bfec92834cc80fba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681736281418 2023-04-17 12:58:03,827 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c7b3aa18e48a4be580c6ffade05a5dfa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681736282100 2023-04-17 12:58:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,827 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3bfa036b61ce4b51ab9f597f7a766461, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283243 2023-04-17 12:58:03,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,828 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 148e1044bb004d72abdf32b52b9fd078, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681736282100 2023-04-17 12:58:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,829 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 27ad2d44db3c42e3ad4b3e2e3f724515, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283245 2023-04-17 12:58:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,840 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,841 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:03,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:03,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,842 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:03,842 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:03,847 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:03,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:03,880 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#202 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:03,942 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:03,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:03,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:03,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:03,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:03,959 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:03,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:03,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:03,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,960 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:03,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:03,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:03,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:03,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:03,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:03,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:03,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:03,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:03,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:03,972 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/6387d302bd724768ae1a6be879413e5f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/6387d302bd724768ae1a6be879413e5f 2023-04-17 12:58:03,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:03,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:03,974 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,974 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:03,975 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:03,975 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,975 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:03,975 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:03,975 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,975 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:03,975 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:03,976 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,976 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,976 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,976 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:03,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:03,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637ff7d4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:03,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:03,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,982 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:03,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:03,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:03,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:03,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:03,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:03,983 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:03,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:03,988 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:03,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:03,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:03,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:03,988 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:03,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/cbebab940c4f43748bcf47d55f1e7081 2023-04-17 12:58:03,991 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,991 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,991 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:03,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736343982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:03,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736343981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,992 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:03,992 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into 6387d302bd724768ae1a6be879413e5f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:03,993 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:03,993 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=12, startTime=1681736283822; duration=0sec 2023-04-17 12:58:03,993 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:03,993 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:03,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,996 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:03,996 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:03,996 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:03,996 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=46.4 K 2023-04-17 12:58:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:03,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:03,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:03,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:03,997 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:03,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736343989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:03,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736343992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:03,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736343992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:03,997 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ac2e6f3f15b346008184a2079405781f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681736281362 2023-04-17 12:58:03,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:03,997 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:03,997 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:03,998 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:04,002 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8fe7072b43b04fdbbcbce239e657c78f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681736281418 2023-04-17 12:58:04,005 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 35b9545b1a254c60a8f707db8f157318, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681736282100 2023-04-17 12:58:04,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:04,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:04,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:04,008 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 92700b65fc23400196992352d8a8e21e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283245 2023-04-17 12:58:04,040 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#205 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:04,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/6662f11951c243978eec8fb0fbb54736 2023-04-17 12:58:04,089 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/6941f74f696341cdbd5bb8d995517583 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/6941f74f696341cdbd5bb8d995517583 2023-04-17 12:58:04,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736344093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736344093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,098 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into 6941f74f696341cdbd5bb8d995517583(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:04,098 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:04,098 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=12, startTime=1681736283823; duration=0sec 2023-04-17 12:58:04,098 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:04,099 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:04,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:04,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:04,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736344099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736344106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736344106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:04,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:04,110 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:04,110 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:04,110 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:04,110 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:04,111 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,111 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:04,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:04,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:04,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:04,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:04,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:04,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:04,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:04,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:04,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:04,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:04,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:04,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:04,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:04,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:04,135 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:04,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:04,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2113e993[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:04,137 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:04,137 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,138 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:04,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:04,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:04,140 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:04,140 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:04,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:04,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:04,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:04,143 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:04,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:04,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:04,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:04,143 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:04,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:04,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,145 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:04,147 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,150 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:04,150 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:04,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:04,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:04,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:04,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,156 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:04,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:04,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:04,159 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:04,159 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:04,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:04,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/1646993384714497827bb7250b15d4eb 2023-04-17 12:58:04,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/cbebab940c4f43748bcf47d55f1e7081 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081 2023-04-17 12:58:04,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081, entries=200, sequenceid=95, filesize=13.9 K 2023-04-17 12:58:04,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/6662f11951c243978eec8fb0fbb54736 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736 2023-04-17 12:58:04,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736, entries=150, sequenceid=95, filesize=11.6 K 2023-04-17 12:58:04,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/1646993384714497827bb7250b15d4eb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb 2023-04-17 12:58:04,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb, entries=150, sequenceid=95, filesize=11.6 K 2023-04-17 12:58:04,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=53.67 KB/54960 for 2a5e7440639effb3958a1a1b6f36fd53 in 250ms, sequenceid=95, compaction requested=false 2023-04-17 12:58:04,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:04,262 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:04,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:04,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:04,270 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:04,270 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:04,270 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:04,270 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:04,271 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,271 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,272 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:04,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,273 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:04,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:04,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:04,273 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,274 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:04,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:04,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:04,274 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:04,274 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:04,274 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:04,279 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:04,279 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:04,279 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:04,279 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:04,280 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:04,280 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:04,280 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:04,280 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:04,280 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:04,282 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:04,282 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:04,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:04,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:04,327 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/55765e0fbc734cd2bd947e7778722278 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/55765e0fbc734cd2bd947e7778722278 2023-04-17 12:58:04,331 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into 55765e0fbc734cd2bd947e7778722278(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:04,332 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:04,332 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=12, startTime=1681736283822; duration=0sec 2023-04-17 12:58:04,332 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:04,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736284324, max timestamp=1681736284330Num uniques 859; 2023-04-17 12:58:04,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:58:04,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736284324, max timestamp=1681736284330Num uniques 859; 2023-04-17 12:58:04,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:58:04,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736284324, max timestamp=1681736284330Num uniques 859; 2023-04-17 12:58:04,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:58:04,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736344346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736344346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736344346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736344347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736344347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736344452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736344452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736344452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736344452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736344452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736344663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736344663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736344665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736344667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736344667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,730 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/111f9007005d4272bb82a978d9d69332 2023-04-17 12:58:04,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736344971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736344974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736344974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736344978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:04,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:04,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736344978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,047 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:58:05,215 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/7d702fb5be8c4972a45d03e7ddc7609b 2023-04-17 12:58:05,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736345482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736345484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736345485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:05,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736345486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:05,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736345490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,679 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a97d13eb6c754f86a4383cba3ee9ebe4 2023-04-17 12:58:05,695 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/111f9007005d4272bb82a978d9d69332 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332 2023-04-17 12:58:05,699 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332, entries=150, sequenceid=108, filesize=11.6 K 2023-04-17 12:58:05,700 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/7d702fb5be8c4972a45d03e7ddc7609b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b 2023-04-17 12:58:05,711 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b, entries=150, sequenceid=108, filesize=11.6 K 2023-04-17 12:58:05,712 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a97d13eb6c754f86a4383cba3ee9ebe4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4 2023-04-17 12:58:05,727 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4, entries=150, sequenceid=108, filesize=11.6 K 2023-04-17 12:58:05,729 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2a5e7440639effb3958a1a1b6f36fd53 in 1450ms, sequenceid=108, compaction requested=true 2023-04-17 12:58:05,729 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:05,729 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:05,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:05,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:05,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:05,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:05,730 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:05,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:05,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:05,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b6b151b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:05,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:05,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,734 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:05,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:05,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:05,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:05,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:05,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:05,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:05,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:05,749 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:05,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:05,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:05,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:05,749 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:05,751 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,751 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,751 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:05,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:05,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:05,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:05,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,752 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:05,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:05,757 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,757 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:05,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:05,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:05,758 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:05,758 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:05,759 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:05,759 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:05,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:05,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:05,858 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:05,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:05,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:05,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:05,866 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:05,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:05,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:05,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:05,870 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,871 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:05,871 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,871 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:05,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:05,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:05,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:05,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:05,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:05,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:05,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:05,876 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:05,876 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:05,876 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:05,876 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:05,877 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:05,877 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:05,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:05,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:05,921 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/7a145c3af83a4e92806401d608d4722f 2023-04-17 12:58:05,980 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d9be0c9f459e4173ba4c05d0f7364483 2023-04-17 12:58:06,035 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/b5ea8030bc4940edbe2debfbf9c0b2e4 2023-04-17 12:58:06,051 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/7a145c3af83a4e92806401d608d4722f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f 2023-04-17 12:58:06,055 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f, entries=150, sequenceid=143, filesize=11.8 K 2023-04-17 12:58:06,056 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d9be0c9f459e4173ba4c05d0f7364483 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483 2023-04-17 12:58:06,061 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483, entries=150, sequenceid=143, filesize=11.8 K 2023-04-17 12:58:06,062 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/b5ea8030bc4940edbe2debfbf9c0b2e4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4 2023-04-17 12:58:06,091 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4, entries=150, sequenceid=143, filesize=11.8 K 2023-04-17 12:58:06,101 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 2a5e7440639effb3958a1a1b6f36fd53 in 225ms, sequenceid=143, compaction requested=true 2023-04-17 12:58:06,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:06,101 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:06,101 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:06,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:06,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:06,114 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:06,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37efa499[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:06,129 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:06,129 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:06,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:06,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:06,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:06,134 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,134 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:06,134 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,134 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:06,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:06,138 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:06,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:06,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:06,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:06,138 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:06,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,150 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,154 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,156 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,159 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:06,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:06,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,160 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:06,160 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:06,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,162 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,261 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:06,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:06,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:06,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:06,287 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:06,287 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,287 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:06,287 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:06,292 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,292 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,313 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,313 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:06,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:06,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:06,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:06,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:06,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,317 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:06,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:06,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:06,324 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,324 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:06,324 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:06,324 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,325 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:06,325 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:06,325 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,325 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:06,325 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:06,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:06,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:06,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb224e7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:06,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,350 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,350 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:06,350 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:06,350 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:06,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:06,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:06,356 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:06,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:06,363 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:06,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:06,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:06,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:06,363 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:06,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:06,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:06,442 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:06,442 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:06,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:06,510 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:06,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:06,546 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:06,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:06,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:06,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:06,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:06,582 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:06,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:06,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:06,587 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:06,590 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:06,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:06,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:06,593 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,593 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:06,593 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:06,593 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:06,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:06,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:06,595 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:06,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15530f73[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:06,596 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:06,596 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736286521, max timestamp=1681736286554Num uniques 859; 2023-04-17 12:58:06,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:06,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:06,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736286521, max timestamp=1681736286546Num uniques 859; 2023-04-17 12:58:06,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:06,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736286521, max timestamp=1681736286546Num uniques 859; 2023-04-17 12:58:06,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:06,602 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:06,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:06,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:06,603 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:06,603 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,603 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,603 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:06,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:06,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:06,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:06,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:06,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:06,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:06,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:06,609 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,610 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,610 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,612 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:06,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,618 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:06,618 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736346627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736346628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736346629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736346630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736346663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,722 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:06,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:06,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:06,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:06,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:06,741 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:06,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:06,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:06,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,744 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:06,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:06,751 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:06,752 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,752 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:06,752 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:06,752 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:06,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,752 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:06,753 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:06,753 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,753 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:06,753 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:06,753 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,753 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:06,753 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:06,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,754 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,755 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,755 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:06,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d3130f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:06,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:06,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,758 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:06,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:06,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:06,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:06,760 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:06,760 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,760 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,760 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736346765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:06,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:06,768 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:06,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:06,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:06,769 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:06,769 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:06,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736346770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,775 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736346770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,784 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736346782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,786 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,786 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:06,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:06,786 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,787 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:06,787 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:06,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,786 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,787 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,787 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,787 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,787 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,788 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736346786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,887 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:06,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:06,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:06,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:06,914 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:06,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:06,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:06,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,915 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:06,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:06,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:06,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:06,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:06,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:06,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:06,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:06,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:06,923 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:06,923 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:06,923 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:06,923 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:06,923 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,923 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:06,923 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:06,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:06,930 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:06,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cd2ae20[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:06,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:06,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:06,969 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:06,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:06,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:06,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:06,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:06,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:06,972 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:06,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:06,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:06,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:06,973 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:06,974 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:06,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,976 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:06,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:06,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:06,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:06,978 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:06,978 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:06,978 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:06,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:06,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:06,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736346970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/feecc5cbadd74a4f9a177fe19a59107d 2023-04-17 12:58:06,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736346978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736346980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:06,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:06,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736346986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736347001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,082 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:07,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:07,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:07,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:07,120 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:07,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:07,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:07,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:07,124 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:07,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:07,124 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,124 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:07,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:07,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:07,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:07,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:07,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:07,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:07,128 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,129 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,129 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:07,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:07,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fd2441[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:07,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:07,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,171 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:07,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:07,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:07,175 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:07,175 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,175 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:07,178 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:07,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:07,188 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:07,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:07,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:07,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:07,188 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:07,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,190 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:07,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:07,203 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:07,203 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:07,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736347281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736347293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736347294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736347294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,303 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:07,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:07,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:07,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736347320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:07,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:07,324 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:07,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:07,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:07,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:07,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:07,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:07,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:07,357 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,357 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:07,357 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:07,357 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:07,357 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:07,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,359 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,359 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,360 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd88f7a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:07,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:07,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:07,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:07,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:07,364 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,364 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:07,365 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:07,365 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,365 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:07,365 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:07,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:07,371 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:07,371 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:07,373 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,373 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,373 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,373 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,375 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,392 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,392 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:07,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,393 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,393 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:07,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/b1f692cd7cab4056aef2326412d07560 2023-04-17 12:58:07,494 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:07,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:07,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:07,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:07,513 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:07,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:07,513 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:07,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,523 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,525 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,528 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:07,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:07,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:07,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:07,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:07,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:07,542 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,542 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:07,542 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:07,542 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:07,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:07,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:07,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:07,547 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:07,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:07,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ddba65[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:07,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:07,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:07,578 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:07,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:07,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:07,582 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:07,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,582 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,582 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:07,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:07,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:07,631 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:07,631 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:07,631 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:07,631 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:07,632 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:07,658 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,658 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,659 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:07,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:07,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:07,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,685 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:07,685 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:07,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,785 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:07,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:07,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:07,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:07,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:07,799 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:07,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:07,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:07,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736347797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,803 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,807 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:07,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:07,808 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:07,816 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,816 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:07,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736347816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736347808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:07,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:07,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736347816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:07,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:07,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:07,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:07,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:07,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736347827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:07,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:07,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:07,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:07,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:07,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:07,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:07,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:07,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,850 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,854 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,854 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:07,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:07,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cf7ab50[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:07,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,898 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:07,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:07,906 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,906 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:07,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:07,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:07,910 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:07,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:07,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:07,914 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:07,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/81a55181ca6f4bb6b5ca5aadd62b12ef 2023-04-17 12:58:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:07,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/feecc5cbadd74a4f9a177fe19a59107d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d 2023-04-17 12:58:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d, entries=150, sequenceid=154, filesize=11.8 K 2023-04-17 12:58:07,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/b1f692cd7cab4056aef2326412d07560 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560 2023-04-17 12:58:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:07,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560, entries=150, sequenceid=154, filesize=11.8 K 2023-04-17 12:58:07,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/81a55181ca6f4bb6b5ca5aadd62b12ef as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef 2023-04-17 12:58:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:07,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:07,992 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:07,992 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:07,992 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:07,992 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:07,992 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:07,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef, entries=150, sequenceid=154, filesize=11.8 K 2023-04-17 12:58:08,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2a5e7440639effb3958a1a1b6f36fd53 in 1491ms, sequenceid=154, compaction requested=true 2023-04-17 12:58:08,001 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:08,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:08,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:08,009 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:08,010 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:58:08,009 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:58:08,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:08,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,021 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:58:08,022 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:08,022 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:08,022 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/55765e0fbc734cd2bd947e7778722278, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=58.5 K 2023-04-17 12:58:08,023 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62208 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:58:08,023 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:08,023 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:08,024 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/6387d302bd724768ae1a6be879413e5f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=60.8 K 2023-04-17 12:58:08,024 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6387d302bd724768ae1a6be879413e5f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283245 2023-04-17 12:58:08,024 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting cbebab940c4f43748bcf47d55f1e7081, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681736283315 2023-04-17 12:58:08,025 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 111f9007005d4272bb82a978d9d69332, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736283974 2023-04-17 12:58:08,025 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 55765e0fbc734cd2bd947e7778722278, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283245 2023-04-17 12:58:08,025 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7a145c3af83a4e92806401d608d4722f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681736284341 2023-04-17 12:58:08,025 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6662f11951c243978eec8fb0fbb54736, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681736283315 2023-04-17 12:58:08,026 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting feecc5cbadd74a4f9a177fe19a59107d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:08,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:08,044 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#216 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:08,046 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7d702fb5be8c4972a45d03e7ddc7609b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736283974 2023-04-17 12:58:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,049 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d9be0c9f459e4173ba4c05d0f7364483, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681736284341 2023-04-17 12:58:08,049 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b1f692cd7cab4056aef2326412d07560, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:08,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:08,050 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:08,080 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#217 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,085 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:08,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:08,089 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:08,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:08,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:08,089 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:08,090 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:08,090 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:08,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:08,106 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/98a457d4f937441cbd98f5db0315e8ee as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/98a457d4f937441cbd98f5db0315e8ee 2023-04-17 12:58:08,108 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/dfd119cc119048b4967e0156dd389208 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/dfd119cc119048b4967e0156dd389208 2023-04-17 12:58:08,138 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into dfd119cc119048b4967e0156dd389208(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:08,138 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:08,138 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=11, startTime=1681736288009; duration=0sec 2023-04-17 12:58:08,138 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:08,138 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:58:08,138 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into 98a457d4f937441cbd98f5db0315e8ee(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:08,142 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:08,142 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=11, startTime=1681736288009; duration=0sec 2023-04-17 12:58:08,142 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:08,159 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:58:08,159 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:08,159 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:08,159 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/6941f74f696341cdbd5bb8d995517583, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=58.5 K 2023-04-17 12:58:08,160 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6941f74f696341cdbd5bb8d995517583, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681736283245 2023-04-17 12:58:08,160 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1646993384714497827bb7250b15d4eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681736283315 2023-04-17 12:58:08,160 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a97d13eb6c754f86a4383cba3ee9ebe4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736283974 2023-04-17 12:58:08,161 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b5ea8030bc4940edbe2debfbf9c0b2e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681736284341 2023-04-17 12:58:08,161 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 81a55181ca6f4bb6b5ca5aadd62b12ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:08,194 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:08,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:08,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:08,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:08,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:08,221 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:08,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:08,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:08,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:08,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:08,227 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:08,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:08,227 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:08,228 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,228 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:08,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:08,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:08,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:08,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:08,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:08,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:08,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:08,232 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#218 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:08,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:08,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:08,233 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:08,233 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:08,233 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:58:08,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:08,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:08,234 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:08,235 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:08,235 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:08,235 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:08,723 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/466aaffac63d459883c0bef39772df6e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/466aaffac63d459883c0bef39772df6e 2023-04-17 12:58:08,724 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fb27dda1299e4af78b990f81f7cb1d03 2023-04-17 12:58:08,779 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into 466aaffac63d459883c0bef39772df6e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:08,779 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:08,779 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=11, startTime=1681736288009; duration=0sec 2023-04-17 12:58:08,779 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:08,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:08,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:08,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736348846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736348846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736348859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736348859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736348862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736348962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736348973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736348974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736348970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:08,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:08,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736348984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736349167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736349178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736349185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,195 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/4c117fcead484c1683af43278a3d32b5 2023-04-17 12:58:09,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736349198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736349200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736349484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736349494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736349494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736349514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:09,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736349522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,680 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8027158deded4506858bfc3300e5c699 2023-04-17 12:58:09,685 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fb27dda1299e4af78b990f81f7cb1d03 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03 2023-04-17 12:58:09,690 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 12:58:09,691 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/4c117fcead484c1683af43278a3d32b5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5 2023-04-17 12:58:09,700 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 12:58:09,701 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8027158deded4506858bfc3300e5c699 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699 2023-04-17 12:58:09,710 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 12:58:09,711 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=60.38 KB/61830 for 2a5e7440639effb3958a1a1b6f36fd53 in 1478ms, sequenceid=193, compaction requested=false 2023-04-17 12:58:09,712 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:09,712 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:09,712 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:09,730 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:09,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,734 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:09,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:09,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@236755a3[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:09,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:09,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:09,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:09,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:09,806 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,806 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:09,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:09,811 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:09,811 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:09,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:09,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:09,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:09,817 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:09,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:09,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:09,817 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:09,817 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:09,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:09,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:09,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:09,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:09,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:09,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:09,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:09,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:09,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:09,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:09,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:09,835 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:09,835 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:09,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:09,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,935 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:09,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:09,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:09,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:09,950 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:09,950 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:09,950 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:09,950 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:09,951 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,951 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:09,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:09,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:09,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:09,953 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:09,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:09,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:09,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:09,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:09,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:09,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:09,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:09,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:09,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:09,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:09,970 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:09,970 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:09,970 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:58:09,970 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:09,971 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:09,971 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:09,971 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:09,972 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:09,972 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:09,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:09,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:10,053 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/733802665b0c486f86f98809c2a61d59 2023-04-17 12:58:10,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736290058, max timestamp=1681736290073Num uniques 859; 2023-04-17 12:58:10,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:58:10,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736290058, max timestamp=1681736290073Num uniques 859; 2023-04-17 12:58:10,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:58:10,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736290058, max timestamp=1681736290073Num uniques 859; 2023-04-17 12:58:10,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:58:10,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736350115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736350118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736350131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736350131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736350133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736350232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736350233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736350238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736350238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736350238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736350442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736350446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736350446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736350452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736350458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,498 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/0210091b6d164c82afb4878b0cc759f7 2023-04-17 12:58:10,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736350750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736350762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736350766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736350766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:10,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736350778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:10,974 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/e6f8a35f561c42ce9f9d3f3f141c46bc 2023-04-17 12:58:10,998 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/733802665b0c486f86f98809c2a61d59 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59 2023-04-17 12:58:11,023 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59, entries=150, sequenceid=206, filesize=11.8 K 2023-04-17 12:58:11,030 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/0210091b6d164c82afb4878b0cc759f7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7 2023-04-17 12:58:11,051 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7, entries=150, sequenceid=206, filesize=11.8 K 2023-04-17 12:58:11,053 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/e6f8a35f561c42ce9f9d3f3f141c46bc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc 2023-04-17 12:58:11,079 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc, entries=150, sequenceid=206, filesize=11.8 K 2023-04-17 12:58:11,090 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2a5e7440639effb3958a1a1b6f36fd53 in 1120ms, sequenceid=206, compaction requested=true 2023-04-17 12:58:11,090 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:11,090 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:11,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:11,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:11,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:11,098 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,100 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:11,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cabadea[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:11,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:11,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,188 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:11,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:11,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:11,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:11,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:11,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:11,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:11,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:11,226 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:11,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:11,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:11,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:11,226 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:11,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,237 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,241 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:11,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,250 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:11,250 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:11,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:11,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:11,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:11,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:11,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736351306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736351310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736351310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736351315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736351316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/55e51087c1124bf0ba407d832c37d9a1 2023-04-17 12:58:11,350 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:11,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:11,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:11,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:11,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:11,370 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:11,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:11,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:11,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:11,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:11,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:11,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:11,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:11,410 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:11,410 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,411 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:11,416 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:11,416 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:11,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:11,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:11,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:11,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:11,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:11,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:11,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736351419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,423 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,424 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,424 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736351419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736351421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736351421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:11,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ad3888[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:11,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736351423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:11,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,426 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:11,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:11,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:11,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:11,438 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,438 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:11,438 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,439 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:11,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:11,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:11,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:11,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:11,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:11,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:11,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/3e8e35453404429ab3059119fd5bf544 2023-04-17 12:58:11,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,476 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:11,477 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:11,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:11,478 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:11,478 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:11,578 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:11,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:11,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:11,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:11,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:11,600 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:11,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:11,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:11,601 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,601 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,606 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:11,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736351622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:11,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736351627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736351627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736351627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736351631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:11,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:11,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:11,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:11,648 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,648 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:11,648 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:11,648 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:11,648 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:11,665 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:11,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d1b666[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:11,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:11,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:11,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:11,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:11,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:11,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:11,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:11,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:11,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:11,695 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:11,695 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:11,695 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:11,695 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:11,696 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:11,697 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,697 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,698 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,700 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,703 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,709 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:11,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:11,709 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:11,709 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,814 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:11,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:11,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:11,835 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:11,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:11,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:11,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,836 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:11,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:11,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:11,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:11,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:11,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:11,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:11,848 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:11,848 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:11,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:11,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:11,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:11,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:11,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:11,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:11,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:11,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,858 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:11,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c022332[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:11,871 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:11,871 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,878 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:11,878 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:11,879 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:11,879 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,879 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:11,881 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,881 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:11,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:11,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:11,890 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:11,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:11,891 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:11,891 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:11,891 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:11,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:11,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:11,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/04aca8ef5d214bc1a7882e0426f9c718 2023-04-17 12:58:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:11,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/55e51087c1124bf0ba407d832c37d9a1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1 2023-04-17 12:58:11,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,924 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:11,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1, entries=150, sequenceid=242, filesize=11.8 K 2023-04-17 12:58:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:11,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/3e8e35453404429ab3059119fd5bf544 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544 2023-04-17 12:58:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736351936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736351942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736351943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736351946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736351946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:11,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:11,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:11,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:11,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:11,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:11,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:11,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:11,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:11,959 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:11,959 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:11,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544, entries=150, sequenceid=242, filesize=11.8 K 2023-04-17 12:58:11,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/04aca8ef5d214bc1a7882e0426f9c718 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718 2023-04-17 12:58:12,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718, entries=150, sequenceid=242, filesize=11.8 K 2023-04-17 12:58:12,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 2a5e7440639effb3958a1a1b6f36fd53 in 754ms, sequenceid=242, compaction requested=true 2023-04-17 12:58:12,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:12,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:12,030 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:12,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:12,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:12,034 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:12,042 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:12,042 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:12,043 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:12,043 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/98a457d4f937441cbd98f5db0315e8ee, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=47.3 K 2023-04-17 12:58:12,053 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:12,054 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:12,054 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:12,054 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/dfd119cc119048b4967e0156dd389208, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=47.3 K 2023-04-17 12:58:12,054 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 98a457d4f937441cbd98f5db0315e8ee, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:12,059 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:12,060 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting dfd119cc119048b4967e0156dd389208, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:12,060 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting fb27dda1299e4af78b990f81f7cb1d03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681736286626 2023-04-17 12:58:12,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:12,062 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4c117fcead484c1683af43278a3d32b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681736286626 2023-04-17 12:58:12,062 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 733802665b0c486f86f98809c2a61d59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681736288837 2023-04-17 12:58:12,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:12,070 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 55e51087c1124bf0ba407d832c37d9a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:12,070 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 0210091b6d164c82afb4878b0cc759f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681736288837 2023-04-17 12:58:12,072 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3e8e35453404429ab3059119fd5bf544, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:12,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:12,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:12,082 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:12,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:12,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:12,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:12,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:12,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:12,100 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:12,106 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,106 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:12,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:12,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:12,113 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:12,113 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,113 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:12,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:12,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:12,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:12,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:12,129 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:12,129 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:12,129 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:12,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:12,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:12,138 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:12,138 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:12,139 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:12,139 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:12,139 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:12,139 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:12,140 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:12,140 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:12,140 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:12,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:12,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:12,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736352502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736352502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736292484, max timestamp=1681736292508Num uniques 859; 2023-04-17 12:58:12,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:58:12,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736292484, max timestamp=1681736292508Num uniques 859; 2023-04-17 12:58:12,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:58:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736292484, max timestamp=1681736292508Num uniques 859; 2023-04-17 12:58:12,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:58:12,569 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/a5aba99e4efc465f984caea969ff8578 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a5aba99e4efc465f984caea969ff8578 2023-04-17 12:58:12,591 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/2dd6ceea22af4f1fa67c4b240086afd0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/2dd6ceea22af4f1fa67c4b240086afd0 2023-04-17 12:58:12,601 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/aa216971ed34440f8642e6c15e60ced4 2023-04-17 12:58:12,605 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into a5aba99e4efc465f984caea969ff8578(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:12,606 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:12,606 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=12, startTime=1681736292030; duration=0sec 2023-04-17 12:58:12,606 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:12,606 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:12,612 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:12,612 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:12,612 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:12,612 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/466aaffac63d459883c0bef39772df6e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=47.3 K 2023-04-17 12:58:12,614 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into 2dd6ceea22af4f1fa67c4b240086afd0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:12,614 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:12,614 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=12, startTime=1681736292034; duration=0sec 2023-04-17 12:58:12,614 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:12,615 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 466aaffac63d459883c0bef39772df6e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681736286497 2023-04-17 12:58:12,618 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8027158deded4506858bfc3300e5c699, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681736286626 2023-04-17 12:58:12,622 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e6f8a35f561c42ce9f9d3f3f141c46bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681736288837 2023-04-17 12:58:12,626 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 04aca8ef5d214bc1a7882e0426f9c718, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:12,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736352640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32806 deadline: 1681736352642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736352658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736352668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736352669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,697 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:12,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736352772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736352774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736352774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736352781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736352794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736352984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736352984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:12,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:12,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736352995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736352998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736353014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,106 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/6ac0147a09a441bf93a1fc6d375f852e 2023-04-17 12:58:13,229 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/3130b0199f0f43fa9021c3107f3cb7c2 2023-04-17 12:58:13,250 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/e516a11022bd446cb9a98e41ed3d51d5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e516a11022bd446cb9a98e41ed3d51d5 2023-04-17 12:58:13,259 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into e516a11022bd446cb9a98e41ed3d51d5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:13,259 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:13,259 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=12, startTime=1681736292034; duration=0sec 2023-04-17 12:58:13,259 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:13,259 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/aa216971ed34440f8642e6c15e60ced4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4 2023-04-17 12:58:13,265 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:58:13,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/6ac0147a09a441bf93a1fc6d375f852e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e 2023-04-17 12:58:13,275 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:58:13,285 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/3130b0199f0f43fa9021c3107f3cb7c2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2 2023-04-17 12:58:13,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736353293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32770 deadline: 1681736353298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32786 deadline: 1681736353307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736353307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,314 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 12:58:13,315 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=132.99 KB/136185 for 2a5e7440639effb3958a1a1b6f36fd53 in 1177ms, sequenceid=253, compaction requested=false 2023-04-17 12:58:13,315 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:13,315 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:13,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:13,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:13,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:13,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:13,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:13,317 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd70ba5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:13,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:13,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:13,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:13,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=139.70 KB heapSize=402.87 KB 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 12:58:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:13,351 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,352 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:13,352 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:13,353 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:13,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:13,383 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:13,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:13,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:13,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:13,383 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:13,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/d5809083513c40bca3fc4805bc8bacd2 2023-04-17 12:58:13,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:13,389 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,409 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:13,409 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:13,410 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:13,409 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:13,409 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,410 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:13,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32814 deadline: 1681736353420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/cba8bc159efa4febae65ebcdec8a0f79 2023-04-17 12:58:13,511 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:13,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:13,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736353528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:13,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:13,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:13,562 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:13,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:13,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:13,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:13,568 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,568 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:13,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:13,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:13,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:13,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:13,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:13,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:13,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:13,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:13,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:13,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:13,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:13,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:13,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:13,595 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:13,602 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,605 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,605 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:13,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@309309ef[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:13,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:13,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,611 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,611 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:13,613 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:13,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:13,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:13,620 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:13,620 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:13,620 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:13,620 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:13,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:13,627 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:13,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:13,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:13,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:13,628 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:13,629 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,630 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,630 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:13,630 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,632 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:13,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:13,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:13,651 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:13,651 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:13,650 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,651 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,652 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:13,761 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:13,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:13,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736353770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:13,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:13,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:13,797 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:13,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:13,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:13,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:13,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736353798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:13,806 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:13,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:13,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:13,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736353807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736353814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:13,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:13,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:13,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:13,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736353818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:13,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:13,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:13,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:13,838 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:13,838 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:13,838 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:13,838 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:13,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:13,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,843 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:13,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e9fb1d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:13,879 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:13,879 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,886 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:13,889 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:13,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:13,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:13,894 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,894 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:13,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:13,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:13,906 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:13,908 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:13,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8a4959021a65471188dc81d66a6ca859 2023-04-17 12:58:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:13,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:13,974 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:13,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:13,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:13,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:13,974 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:14,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:14,007 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:14,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:14,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/d5809083513c40bca3fc4805bc8bacd2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2 2023-04-17 12:58:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:14,022 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:14,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:14,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:14,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:14,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2, entries=200, sequenceid=294, filesize=14.3 K 2023-04-17 12:58:14,056 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:14,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/cba8bc159efa4febae65ebcdec8a0f79 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79 2023-04-17 12:58:14,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736354076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:14,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,103 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:14,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:14,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:14,104 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:14,104 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:14,104 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:14,104 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:14,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:14,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:14,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79, entries=150, sequenceid=294, filesize=11.9 K 2023-04-17 12:58:14,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:14,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:14,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/8a4959021a65471188dc81d66a6ca859 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859 2023-04-17 12:58:14,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859, entries=150, sequenceid=294, filesize=11.9 K 2023-04-17 12:58:14,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~139.70 KB/143055, heapSize ~402.82 KB/412488, currentSize=40.25 KB/41220 for 2a5e7440639effb3958a1a1b6f36fd53 in 827ms, sequenceid=294, compaction requested=true 2023-04-17 12:58:14,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:14,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:14,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:14,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:58:14,161 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:14,161 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:14,173 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39142 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:14,173 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:14,174 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:14,174 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a5aba99e4efc465f984caea969ff8578, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=38.2 K 2023-04-17 12:58:14,174 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:14,174 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:14,175 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:14,175 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e516a11022bd446cb9a98e41ed3d51d5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=35.8 K 2023-04-17 12:58:14,182 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e516a11022bd446cb9a98e41ed3d51d5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:14,190 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3130b0199f0f43fa9021c3107f3cb7c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736291300 2023-04-17 12:58:14,191 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a5aba99e4efc465f984caea969ff8578, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:14,194 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting aa216971ed34440f8642e6c15e60ced4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736291300 2023-04-17 12:58:14,198 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d5809083513c40bca3fc4805bc8bacd2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:14,198 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8a4959021a65471188dc81d66a6ca859, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:14,208 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:14,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:14,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:14,233 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:14,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,235 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,235 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:14,238 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:14,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:14,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:14,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:14,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:14,243 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:14,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:14,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:14,251 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:14,251 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:14,251 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:14,253 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:14,253 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:14,253 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:14,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:14,267 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:14,267 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 12:58:14,267 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:14,268 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:14,268 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:14,268 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:14,268 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:14,268 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:14,318 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/d49e200ae016408da2f838f6d94b9ae9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/d49e200ae016408da2f838f6d94b9ae9 2023-04-17 12:58:14,368 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into d49e200ae016408da2f838f6d94b9ae9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:14,368 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:14,368 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=13, startTime=1681736294158; duration=0sec 2023-04-17 12:58:14,368 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:14,368 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:14,378 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/e6585e9d4649468ab485d924d387ba5f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e6585e9d4649468ab485d924d387ba5f 2023-04-17 12:58:14,382 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:14,383 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:14,383 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:14,383 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/2dd6ceea22af4f1fa67c4b240086afd0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=35.8 K 2023-04-17 12:58:14,383 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2dd6ceea22af4f1fa67c4b240086afd0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681736290115 2023-04-17 12:58:14,384 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6ac0147a09a441bf93a1fc6d375f852e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681736291300 2023-04-17 12:58:14,387 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting cba8bc159efa4febae65ebcdec8a0f79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:14,410 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into e6585e9d4649468ab485d924d387ba5f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:14,410 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:14,410 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=13, startTime=1681736294158; duration=0sec 2023-04-17 12:58:14,410 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:14,415 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:14,475 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/c9a70c49d2a343d09b37e04b48c18fde as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/c9a70c49d2a343d09b37e04b48c18fde 2023-04-17 12:58:14,494 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into c9a70c49d2a343d09b37e04b48c18fde(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:14,494 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:14,494 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=13, startTime=1681736294158; duration=0sec 2023-04-17 12:58:14,494 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:14,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:14,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736294705, max timestamp=1681736294757Num uniques 859; 2023-04-17 12:58:14,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:14,786 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/5e36aaaff54347d4ae2a662de6af8479 2023-04-17 12:58:14,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736354835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736354838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736354840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736354841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736354842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,850 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/e35b5fbe46f54b22978d1536d4fe9714 2023-04-17 12:58:14,889 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/dada9bd09c9148b68fcfe0e26a8beabe 2023-04-17 12:58:14,897 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/5e36aaaff54347d4ae2a662de6af8479 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479 2023-04-17 12:58:14,908 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 12:58:14,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/e35b5fbe46f54b22978d1536d4fe9714 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714 2023-04-17 12:58:14,942 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 12:58:14,943 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/dada9bd09c9148b68fcfe0e26a8beabe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe 2023-04-17 12:58:14,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736354944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736354944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:14,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736354948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,960 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 12:58:14,961 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=136.42 KB/139690 for 2a5e7440639effb3958a1a1b6f36fd53 in 694ms, sequenceid=303, compaction requested=false 2023-04-17 12:58:14,961 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:14,961 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:14,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:14,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:14,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:14,961 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:14,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:14,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:14,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:14,971 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:14,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:14,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356f4681[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:14,991 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:14,991 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,002 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:15,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:15,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:15,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:15,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:15,014 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:15,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:15,018 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:15,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:15,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:15,018 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:15,018 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:15,040 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,040 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,042 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,108 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,110 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,110 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,110 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,111 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:15,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:15,112 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:15,112 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:15,110 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:15,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=149.83 KB heapSize=441.41 KB 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:15,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:15,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736355175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736355180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736355182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,213 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:15,222 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fd7074ff62d04992ad0ec66a2bfc5533 2023-04-17 12:58:15,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:15,258 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:15,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:15,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:15,262 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,262 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,266 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,270 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:15,290 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:15,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736355291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:15,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:15,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:15,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:15,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736355283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736355300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:15,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:15,310 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,310 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:15,310 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:15,310 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:15,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:15,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:15,310 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:15,320 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,320 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/93fe598cc9d4408c83fb20d11da6f792 2023-04-17 12:58:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:15,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c0e9aa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:15,355 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:15,355 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,362 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:15,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:15,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:15,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:15,376 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:15,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:15,383 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:15,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:15,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:15,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:15,383 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:15,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:15,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:15,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,392 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/87a8524f4b8d49fa8b8ecd0038672fc0 2023-04-17 12:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,395 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,397 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,397 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,399 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,399 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,399 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:15,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:15,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fd7074ff62d04992ad0ec66a2bfc5533 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533 2023-04-17 12:58:15,401 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:15,401 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:15,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533, entries=250, sequenceid=336, filesize=16.7 K 2023-04-17 12:58:15,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/93fe598cc9d4408c83fb20d11da6f792 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792 2023-04-17 12:58:15,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792, entries=150, sequenceid=336, filesize=11.9 K 2023-04-17 12:58:15,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/87a8524f4b8d49fa8b8ecd0038672fc0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0 2023-04-17 12:58:15,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0, entries=150, sequenceid=336, filesize=11.9 K 2023-04-17 12:58:15,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~441.36 KB/451952, currentSize=26.84 KB/27480 for 2a5e7440639effb3958a1a1b6f36fd53 in 302ms, sequenceid=336, compaction requested=true 2023-04-17 12:58:15,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:15,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:15,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:15,462 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:15,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:15,464 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41984 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:15,464 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:15,464 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,464 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e6585e9d4649468ab485d924d387ba5f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=41 K 2023-04-17 12:58:15,466 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e6585e9d4649468ab485d924d387ba5f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:15,466 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:15,466 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5e36aaaff54347d4ae2a662de6af8479, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681736293337 2023-04-17 12:58:15,467 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting fd7074ff62d04992ad0ec66a2bfc5533, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294795 2023-04-17 12:58:15,474 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:15,474 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:15,474 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,474 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/c9a70c49d2a343d09b37e04b48c18fde, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=36.2 K 2023-04-17 12:58:15,476 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c9a70c49d2a343d09b37e04b48c18fde, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:15,476 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e35b5fbe46f54b22978d1536d4fe9714, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681736293337 2023-04-17 12:58:15,481 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 93fe598cc9d4408c83fb20d11da6f792, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294819 2023-04-17 12:58:15,502 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:15,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:15,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:15,523 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:15,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:15,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:15,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:15,539 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:15,542 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:15,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:15,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:15,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:15,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:15,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:15,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:15,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:15,568 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:15,568 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,568 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:15,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:15,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:15,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:15,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:15,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:15,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:15,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:15,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:15,606 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,607 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,607 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,607 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:15,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3834462c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:15,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:15,609 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,609 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:15,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:15,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:15,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:15,611 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:15,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:15,613 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:15,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:15,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:15,613 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:15,613 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:15,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,707 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:15,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:15,708 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:15,708 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:15,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,708 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,708 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,708 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,708 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,708 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,707 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:15,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/baafe55cfb574547b7373660c381d148 2023-04-17 12:58:15,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736355764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681736295679, max timestamp=1681736295766Num uniques 859; 2023-04-17 12:58:15,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 12:58:15,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736295698, max timestamp=1681736295766Num uniques 859; 2023-04-17 12:58:15,770 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:15,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736295698, max timestamp=1681736295766Num uniques 859; 2023-04-17 12:58:15,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:15,808 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:15,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:15,820 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/445e8d5f89224eb69ac4b79b218d0a05 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/445e8d5f89224eb69ac4b79b218d0a05 2023-04-17 12:58:15,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:15,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:15,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:15,858 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:15,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:15,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:15,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,874 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:15,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:15,877 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,898 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:15,901 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into 445e8d5f89224eb69ac4b79b218d0a05(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:15,901 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:15,901 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=13, startTime=1681736295462; duration=0sec 2023-04-17 12:58:15,901 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:15,901 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:15,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:15,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:15,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:15,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:15,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:15,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:15,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:15,914 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:15,914 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:15,915 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,915 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/d49e200ae016408da2f838f6d94b9ae9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=36.2 K 2023-04-17 12:58:15,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:15,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:15,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:15,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:15,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:15,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:15,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:15,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:15,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:15,928 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d49e200ae016408da2f838f6d94b9ae9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681736292624 2023-04-17 12:58:15,934 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting dada9bd09c9148b68fcfe0e26a8beabe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681736293337 2023-04-17 12:58:15,935 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 87a8524f4b8d49fa8b8ecd0038672fc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294819 2023-04-17 12:58:15,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736355926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736355924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:15,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736355939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,948 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,966 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe26324[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:15,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:15,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:15,980 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:15,980 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:15,980 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:15,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:15,982 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:15,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:15,985 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:15,985 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:15,985 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:15,985 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:15,985 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:15,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:15,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:15,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:15,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:15,991 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:15,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:15,995 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:15,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:15,996 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/4ac5adc129e04663b2992e5ff0ff99a4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4ac5adc129e04663b2992e5ff0ff99a4 2023-04-17 12:58:15,997 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:15,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,005 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into 4ac5adc129e04663b2992e5ff0ff99a4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:16,005 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:16,005 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=13, startTime=1681736295462; duration=0sec 2023-04-17 12:58:16,005 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:16,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:16,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:16,010 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,011 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:16,011 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:16,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,011 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736356047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736356049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736356049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,111 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:16,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:16,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:16,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:16,127 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:16,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:16,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:16,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,141 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,141 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:16,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:16,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:16,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:16,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:16,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:16,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:16,167 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,167 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:16,167 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:16,167 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:16,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:16,179 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:16,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f66d17[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:16,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:16,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:16,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:16,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:16,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:16,199 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:16,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:16,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:16,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:16,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:16,201 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:16,201 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:16,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,208 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:16,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:16,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,208 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,208 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:16,208 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:16,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736356252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736356262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736356262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d8170d7b931a473d99c1a54f761ed294 2023-04-17 12:58:16,312 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:16,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:16,343 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:16,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:16,343 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:16,346 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,346 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,347 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,348 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,348 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:16,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:16,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:16,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:16,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:16,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:16,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/9fc2cd6d9464442387cbd68cbce33218 2023-04-17 12:58:16,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/baafe55cfb574547b7373660c381d148 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148 2023-04-17 12:58:16,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148, entries=150, sequenceid=347, filesize=11.9 K 2023-04-17 12:58:16,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d8170d7b931a473d99c1a54f761ed294 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294 2023-04-17 12:58:16,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:16,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:16,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:16,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:16,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:16,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:16,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:16,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:16,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294, entries=150, sequenceid=347, filesize=11.9 K 2023-04-17 12:58:16,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/9fc2cd6d9464442387cbd68cbce33218 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218 2023-04-17 12:58:16,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218, entries=150, sequenceid=347, filesize=11.9 K 2023-04-17 12:58:16,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 2a5e7440639effb3958a1a1b6f36fd53 in 876ms, sequenceid=347, compaction requested=false 2023-04-17 12:58:16,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:16,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c4b1a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:16,410 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:16,410 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:16,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:16,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:16,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:16,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:16,435 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:16,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:16,442 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:16,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:16,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:16,442 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:16,442 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:16,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,500 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a04069774d6f4761bd07398e90a1abfc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a04069774d6f4761bd07398e90a1abfc 2023-04-17 12:58:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,518 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,527 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into a04069774d6f4761bd07398e90a1abfc(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:16,527 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:16,527 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=13, startTime=1681736295462; duration=0sec 2023-04-17 12:58:16,527 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:16,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:16,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:16,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:16,586 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,586 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:16,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:16,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:16,591 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:16,591 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:16,592 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,592 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,593 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,593 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,593 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,593 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,593 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736356628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736356628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736356629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,691 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:16,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:16,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:16,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:16,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:16,714 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:16,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:16,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:16,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736356733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736356734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736356742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:16,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:16,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:16,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:16,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:16,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:16,778 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:16,782 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,782 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:16,783 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:16,783 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:16,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:16,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:16,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:16,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:16,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,786 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:16,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,793 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:16,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:16,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df378fd[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:16,823 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:16,823 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,839 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:16,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:16,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:16,846 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,846 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:16,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:32784 deadline: 1681736356847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:16,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:16,852 DEBUG [Thread-1491] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:58:16,862 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736356870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,889 DEBUG [Thread-1497] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4224 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:58:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:16,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:16,922 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:16,922 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:16,922 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:16,922 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:16,922 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:16,931 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,931 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,934 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,936 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:16,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:16,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:16,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:16,940 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:16,940 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:16,940 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:16,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:16,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736356941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:16,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:16,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736356941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:16,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:16,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736356963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,042 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:17,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:17,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:17,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:17,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:17,051 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:17,051 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:17,051 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:17,051 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:17,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/1265553e5e1c4b55b521669f787deb8b 2023-04-17 12:58:17,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,072 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:17,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:17,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:17,077 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,082 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:17,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:17,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:17,084 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:17,084 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:17,084 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:17,106 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:17,106 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:17,106 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:17,106 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:17,106 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:17,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,130 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:17,134 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:17,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/51c92b279d874f268e2b0c1eb1e5d6b2 2023-04-17 12:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:17,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:17,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51c4c324[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:17,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:17,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,185 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:17,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:17,185 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:17,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:17,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:17,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:17,207 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:17,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:17,214 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:17,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:17,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:17,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:17,214 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:17,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:17,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:17,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/ad7b824060a846efbf4ad4430a6eae6f 2023-04-17 12:58:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736357251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:17,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:17,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:17,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736357259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/1265553e5e1c4b55b521669f787deb8b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b 2023-04-17 12:58:17,266 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736357271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b, entries=200, sequenceid=386, filesize=14.3 K 2023-04-17 12:58:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:17,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/51c92b279d874f268e2b0c1eb1e5d6b2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2 2023-04-17 12:58:17,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:17,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2, entries=150, sequenceid=386, filesize=11.9 K 2023-04-17 12:58:17,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/ad7b824060a846efbf4ad4430a6eae6f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f 2023-04-17 12:58:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:17,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:17,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:17,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:17,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:17,356 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,356 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,357 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:17,357 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:17,362 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:17,366 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:17,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f, entries=150, sequenceid=386, filesize=11.9 K 2023-04-17 12:58:17,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:17,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 2a5e7440639effb3958a1a1b6f36fd53 in 800ms, sequenceid=386, compaction requested=true 2023-04-17 12:58:17,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:17,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:17,374 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:17,378 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:17,374 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:17,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:17,382 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:17,382 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:17,382 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:17,383 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4ac5adc129e04663b2992e5ff0ff99a4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=38.7 K 2023-04-17 12:58:17,383 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4ac5adc129e04663b2992e5ff0ff99a4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294819 2023-04-17 12:58:17,386 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting baafe55cfb574547b7373660c381d148, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681736295505 2023-04-17 12:58:17,387 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1265553e5e1c4b55b521669f787deb8b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:17,387 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:17,387 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:17,387 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:17,387 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/445e8d5f89224eb69ac4b79b218d0a05, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=36.3 K 2023-04-17 12:58:17,398 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 445e8d5f89224eb69ac4b79b218d0a05, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294819 2023-04-17 12:58:17,402 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d8170d7b931a473d99c1a54f761ed294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681736295505 2023-04-17 12:58:17,404 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 51c92b279d874f268e2b0c1eb1e5d6b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:17,440 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:17,458 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:17,470 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:17,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:17,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:17,498 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:17,498 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:17,499 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:17,499 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:17,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,518 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:17,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:17,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:17,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:17,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:17,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:17,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:17,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:17,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:17,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:17,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:17,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:17,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:17,538 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:17,539 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:17,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:17,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:17,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681736297806, max timestamp=1681736297835Num uniques 859; 2023-04-17 12:58:17,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 12:58:17,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736297806, max timestamp=1681736297835Num uniques 859; 2023-04-17 12:58:17,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:17,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736297806, max timestamp=1681736297835Num uniques 859; 2023-04-17 12:58:17,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:17,926 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/4f7f60eeb1c146738ef9736d6920dd56 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4f7f60eeb1c146738ef9736d6920dd56 2023-04-17 12:58:17,935 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/4b651e69058a45eeb7d0174d3fbc784d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4b651e69058a45eeb7d0174d3fbc784d 2023-04-17 12:58:18,014 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into 4b651e69058a45eeb7d0174d3fbc784d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:18,014 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:18,014 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=13, startTime=1681736297374; duration=0sec 2023-04-17 12:58:18,015 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:18,015 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:18,016 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into 4f7f60eeb1c146738ef9736d6920dd56(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:18,016 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:18,016 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=13, startTime=1681736297374; duration=0sec 2023-04-17 12:58:18,016 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:18,017 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/285fa4a3704540028206a16593be1d39 2023-04-17 12:58:18,018 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:18,018 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:18,018 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:18,018 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a04069774d6f4761bd07398e90a1abfc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=36.3 K 2023-04-17 12:58:18,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736357998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,030 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a04069774d6f4761bd07398e90a1abfc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681736294819 2023-04-17 12:58:18,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736358016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,034 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9fc2cd6d9464442387cbd68cbce33218, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681736295505 2023-04-17 12:58:18,035 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ad7b824060a846efbf4ad4430a6eae6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736358026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,079 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:18,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736358126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736358138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736358146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736358332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736358355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736358365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,545 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/a96517f12c9b47b6bbf4d3a9fb6694bb 2023-04-17 12:58:18,554 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/f68f51c461db438cacaaae9636feb8b8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/f68f51c461db438cacaaae9636feb8b8 2023-04-17 12:58:18,588 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into f68f51c461db438cacaaae9636feb8b8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:18,588 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:18,588 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=13, startTime=1681736297378; duration=0sec 2023-04-17 12:58:18,588 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:18,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736358642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736358662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:18,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:18,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736358690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,036 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a3f718dfac754ec69ebaf21c976b2d79 2023-04-17 12:58:19,098 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/285fa4a3704540028206a16593be1d39 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39 2023-04-17 12:58:19,125 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39, entries=150, sequenceid=396, filesize=11.9 K 2023-04-17 12:58:19,134 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/a96517f12c9b47b6bbf4d3a9fb6694bb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb 2023-04-17 12:58:19,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736359154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:19,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736359170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,182 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb, entries=150, sequenceid=396, filesize=11.9 K 2023-04-17 12:58:19,198 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a3f718dfac754ec69ebaf21c976b2d79 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79 2023-04-17 12:58:19,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:19,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736359214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,242 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79, entries=150, sequenceid=396, filesize=11.9 K 2023-04-17 12:58:19,244 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2a5e7440639effb3958a1a1b6f36fd53 in 1706ms, sequenceid=396, compaction requested=false 2023-04-17 12:58:19,244 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:19,244 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:19,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:19,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:19,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:19,245 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:19,250 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:19,254 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,254 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:19,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@727992de[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:19,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:19,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,313 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:19,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:19,318 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:19,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:19,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:19,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:19,326 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:19,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:19,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:19,401 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:19,401 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:19,401 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:19,401 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:19,401 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:19,414 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,414 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:19,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:19,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:19,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:19,468 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:19,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:19,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:19,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:19,468 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:19,468 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:19,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:19,471 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:19,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:19,570 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:19,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:19,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:19,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:19,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:19,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:19,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:19,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:19,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:19,612 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,612 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:19,615 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:19,615 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:19,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:19,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:19,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:19,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:19,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:19,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:19,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:19,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:19,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:19,631 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:19,631 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:19,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:19,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:19,634 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 12:58:19,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:19,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:19,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:19,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:19,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:19,635 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:20,129 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fadeea53cc194c798b6ec2ff6a8db92d 2023-04-17 12:58:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:20,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:20,196 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/ee821da794434a7aa37cfb3caa9e65f7 2023-04-17 12:58:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736360211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736360211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32814 deadline: 1681736360231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,271 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/b04616ade12f4015aa2da2312f4c4ac8 2023-04-17 12:58:20,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736360319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736360319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,322 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/fadeea53cc194c798b6ec2ff6a8db92d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d 2023-04-17 12:58:20,375 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d, entries=150, sequenceid=436, filesize=11.9 K 2023-04-17 12:58:20,390 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/ee821da794434a7aa37cfb3caa9e65f7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7 2023-04-17 12:58:20,427 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7, entries=150, sequenceid=436, filesize=11.9 K 2023-04-17 12:58:20,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/b04616ade12f4015aa2da2312f4c4ac8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8 2023-04-17 12:58:20,454 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8, entries=150, sequenceid=436, filesize=11.9 K 2023-04-17 12:58:20,456 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 2a5e7440639effb3958a1a1b6f36fd53 in 822ms, sequenceid=436, compaction requested=true 2023-04-17 12:58:20,456 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:20,456 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:20,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:20,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:20,456 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:20,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:20,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:20,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17a82310[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:20,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:20,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,487 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:20,488 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:20,488 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:20,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:20,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:20,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:20,494 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,494 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:20,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:20,515 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:20,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:20,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:20,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:20,515 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:20,520 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:20,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,522 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,524 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:20,526 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:20,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:20,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:20,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:20,530 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:20,531 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:20,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:20,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:58:20,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:20,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:20,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:20,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:20,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:20,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:20,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:20,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/477137be215a464f9140b86721931885 2023-04-17 12:58:20,631 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:20,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:20,650 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:20,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:20,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:20,650 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:20,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300560, max timestamp=1681736300647Num uniques 859; 2023-04-17 12:58:20,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:20,655 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300560, max timestamp=1681736300650Num uniques 859; 2023-04-17 12:58:20,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:20,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300560, max timestamp=1681736300647Num uniques 859; 2023-04-17 12:58:20,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:20,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,667 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:20,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:20,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:20,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:20,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:20,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:20,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:20,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:20,679 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:20,679 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:20,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:20,680 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:20,680 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:20,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:20,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:20,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/538e581bee904cf0929bc9a22cffb105 2023-04-17 12:58:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:20,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@111a8588[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:20,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:20,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,730 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:20,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:20,731 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:20,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,733 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:20,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:20,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:20,736 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:20,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/89bfdc947d3642ff8d2c2cd877136f98 2023-04-17 12:58:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/477137be215a464f9140b86721931885 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885 2023-04-17 12:58:20,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681736300748, max timestamp=1681736300802Num uniques 1018; 2023-04-17 12:58:20,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 12:58:20,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736300765, max timestamp=1681736300802Num uniques 1018; 2023-04-17 12:58:20,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:58:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736300765, max timestamp=1681736300802Num uniques 1018; 2023-04-17 12:58:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:58:20,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885, entries=150, sequenceid=446, filesize=11.9 K 2023-04-17 12:58:20,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:20,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:20,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:20,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:20,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:20,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:20,822 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:20,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/538e581bee904cf0929bc9a22cffb105 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105 2023-04-17 12:58:20,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105, entries=100, sequenceid=446, filesize=9.5 K 2023-04-17 12:58:20,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/89bfdc947d3642ff8d2c2cd877136f98 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98 2023-04-17 12:58:20,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:20,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98, entries=100, sequenceid=446, filesize=9.5 K 2023-04-17 12:58:20,847 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=62.62 KB/64120 for 2a5e7440639effb3958a1a1b6f36fd53 in 315ms, sequenceid=446, compaction requested=true 2023-04-17 12:58:20,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:20,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:20,847 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:20,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:20,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:20,848 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:20,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=69.33 KB heapSize=190 KB 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:20,848 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:20,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:20,850 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:20,850 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/B is initiating minor compaction (all files) 2023-04-17 12:58:20,850 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/B in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:20,850 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4b651e69058a45eeb7d0174d3fbc784d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=45.9 K 2023-04-17 12:58:20,851 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4b651e69058a45eeb7d0174d3fbc784d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:20,851 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a96517f12c9b47b6bbf4d3a9fb6694bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681736296587 2023-04-17 12:58:20,852 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49490 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:20,852 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/A is initiating minor compaction (all files) 2023-04-17 12:58:20,852 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/A in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:20,852 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4f7f60eeb1c146738ef9736d6920dd56, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=48.3 K 2023-04-17 12:58:20,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:20,856 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4f7f60eeb1c146738ef9736d6920dd56, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:20,857 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ee821da794434a7aa37cfb3caa9e65f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681736297927 2023-04-17 12:58:20,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 285fa4a3704540028206a16593be1d39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681736296587 2023-04-17 12:58:20,857 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting fadeea53cc194c798b6ec2ff6a8db92d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681736297927 2023-04-17 12:58:20,858 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 477137be215a464f9140b86721931885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681736300184 2023-04-17 12:58:20,863 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 538e581bee904cf0929bc9a22cffb105, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681736300184 2023-04-17 12:58:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,898 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#A#compaction#268 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,906 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:20,915 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#B#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/4cd2c144c5b440d9acbb64f24424b744 2023-04-17 12:58:20,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:20,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:20,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32784 deadline: 1681736360923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736360924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736360926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:20,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736360928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:20,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:20,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:20,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:20,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:20,944 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:20,944 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:20,995 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/4e60ed2915f84bffa76bb7d051482f79 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4e60ed2915f84bffa76bb7d051482f79 2023-04-17 12:58:21,009 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/A of 2a5e7440639effb3958a1a1b6f36fd53 into 4e60ed2915f84bffa76bb7d051482f79(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:21,009 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:21,009 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/A, priority=12, startTime=1681736300847; duration=0sec 2023-04-17 12:58:21,009 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:21,009 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:21,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:21,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 2a5e7440639effb3958a1a1b6f36fd53/C is initiating minor compaction (all files) 2023-04-17 12:58:21,012 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a5e7440639effb3958a1a1b6f36fd53/C in TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:21,012 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/f68f51c461db438cacaaae9636feb8b8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp, totalSize=45.9 K 2023-04-17 12:58:21,012 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f68f51c461db438cacaaae9636feb8b8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681736295914 2023-04-17 12:58:21,013 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a3f718dfac754ec69ebaf21c976b2d79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681736296587 2023-04-17 12:58:21,013 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b04616ade12f4015aa2da2312f4c4ac8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681736297927 2023-04-17 12:58:21,014 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 89bfdc947d3642ff8d2c2cd877136f98, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681736300184 2023-04-17 12:58:21,025 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a5e7440639effb3958a1a1b6f36fd53#C#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:21,044 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:21,047 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/d74c8b2b2f294d11a90e77da235e5bd9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d74c8b2b2f294d11a90e77da235e5bd9 2023-04-17 12:58:21,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736361037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736361038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32784 deadline: 1681736361038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736361038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:21,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/B of 2a5e7440639effb3958a1a1b6f36fd53 into d74c8b2b2f294d11a90e77da235e5bd9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:21,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:21,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/B, priority=12, startTime=1681736300847; duration=0sec 2023-04-17 12:58:21,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:21,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:21,083 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:21,083 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:21,083 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:21,083 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:21,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:21,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/e2ce8f10bd464d51ac36d105ae30aedd 2023-04-17 12:58:21,091 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:21,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:21,094 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:21,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:21,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:21,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:21,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:21,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:21,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:21,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:21,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:21,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:21,102 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:21,103 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:21,103 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. started... 2023-04-17 12:58:21,103 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. as already flushing 2023-04-17 12:58:21,103 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:21,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:21,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:21,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:21,103 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:21,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,110 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:21,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,114 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:21,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:21,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6630a96c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:21,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:21,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,146 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:21,146 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:21,146 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:21,147 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:21,147 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:21,147 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:21,154 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,158 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:21,158 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:21,158 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:21,158 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:21,158 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:21,158 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:21,158 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:21,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,170 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:21,171 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:21,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,194 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:21,204 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:21,204 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:21,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:21,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:21,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:21,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:21,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:21,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:21,206 DEBUG [Thread-1501] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:21,206 DEBUG [Thread-1501] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:21,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/e51ee4f8a2d041aa9bb04f0b5bddf694 2023-04-17 12:58:21,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/4cd2c144c5b440d9acbb64f24424b744 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4cd2c144c5b440d9acbb64f24424b744 2023-04-17 12:58:21,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32770 deadline: 1681736361263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32806 deadline: 1681736361263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32786 deadline: 1681736361264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:21,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:32784 deadline: 1681736361268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:21,292 DEBUG [Thread-1510] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d22a55e to 127.0.0.1:62647 2023-04-17 12:58:21,292 DEBUG [Thread-1510] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,294 DEBUG [Thread-1502] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:62647 2023-04-17 12:58:21,294 DEBUG [Thread-1502] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,295 DEBUG [Thread-1508] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:62647 2023-04-17 12:58:21,295 DEBUG [Thread-1508] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,298 DEBUG [Thread-1504] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:62647 2023-04-17 12:58:21,298 DEBUG [Thread-1504] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4cd2c144c5b440d9acbb64f24424b744, entries=200, sequenceid=490, filesize=14.3 K 2023-04-17 12:58:21,300 DEBUG [Thread-1506] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:62647 2023-04-17 12:58:21,300 DEBUG [Thread-1506] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/e2ce8f10bd464d51ac36d105ae30aedd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e2ce8f10bd464d51ac36d105ae30aedd 2023-04-17 12:58:21,306 DEBUG [Thread-1501] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:21,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:21,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e2ce8f10bd464d51ac36d105ae30aedd, entries=150, sequenceid=490, filesize=11.9 K 2023-04-17 12:58:21,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/e51ee4f8a2d041aa9bb04f0b5bddf694 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e51ee4f8a2d041aa9bb04f0b5bddf694 2023-04-17 12:58:21,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e51ee4f8a2d041aa9bb04f0b5bddf694, entries=150, sequenceid=490, filesize=11.9 K 2023-04-17 12:58:21,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~189.95 KB/194512, currentSize=114.05 KB/116790 for 2a5e7440639effb3958a1a1b6f36fd53 in 466ms, sequenceid=490, compaction requested=false 2023-04-17 12:58:21,315 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:21,513 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/fa9cf1cb2091499b94d737982c840848 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/fa9cf1cb2091499b94d737982c840848 2023-04-17 12:58:21,517 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a5e7440639effb3958a1a1b6f36fd53/C of 2a5e7440639effb3958a1a1b6f36fd53 into fa9cf1cb2091499b94d737982c840848(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:21,517 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:21,517 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53., storeName=2a5e7440639effb3958a1a1b6f36fd53/C, priority=12, startTime=1681736300847; duration=0sec 2023-04-17 12:58:21,517 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300908, max timestamp=1681736301570Num uniques 859; 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300908, max timestamp=1681736301570Num uniques 859; 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736300908, max timestamp=1681736301570Num uniques 859; 2023-04-17 12:58:21,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:21,574 DEBUG [Thread-1499] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a8a575d to 127.0.0.1:62647 2023-04-17 12:58:21,574 DEBUG [Thread-1495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:62647 2023-04-17 12:58:21,574 DEBUG [Thread-1495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,574 DEBUG [Thread-1499] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,576 DEBUG [Thread-1497] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:62647 2023-04-17 12:58:21,577 DEBUG [Thread-1497] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:21,578 DEBUG [Thread-1491] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7117e4df to 127.0.0.1:62647 2023-04-17 12:58:21,578 DEBUG [Thread-1491] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:22,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:22,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a5e7440639effb3958a1a1b6f36fd53 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-17 12:58:22,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=A 2023-04-17 12:58:22,244 DEBUG [Thread-1493] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:62647 2023-04-17 12:58:22,245 DEBUG [Thread-1493] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:22,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:22,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=B 2023-04-17 12:58:22,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:22,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a5e7440639effb3958a1a1b6f36fd53, store=C 2023-04-17 12:58:22,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-17 12:58:22,245 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 1987 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2048 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2017 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2060 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2007 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:58:22,246 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:58:22,246 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64e0df67 to 127.0.0.1:62647 2023-04-17 12:58:22,246 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:22,248 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:58:22,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:58:22,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:22,262 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736302262"}]},"ts":"1681736302262"} 2023-04-17 12:58:22,265 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:58:22,349 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:58:22,350 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, UNASSIGN}] 2023-04-17 12:58:22,351 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, UNASSIGN 2023-04-17 12:58:22,351 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2a5e7440639effb3958a1a1b6f36fd53, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:22,352 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736302351"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736302351"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736302351"}]},"ts":"1681736302351"} 2023-04-17 12:58:22,353 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:22,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:22,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 2a5e7440639effb3958a1a1b6f36fd53, disabling compactions & flushes 2023-04-17 12:58:22,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:22,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:22,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/e5bbc61e5a97426b808bc1911ff6d51b 2023-04-17 12:58:22,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:23,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/82b43f08b00844709ce0a7e999420b43 2023-04-17 12:58:23,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:23,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a87188f3280845f4beb85fd14bf06201 2023-04-17 12:58:23,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/A/e5bbc61e5a97426b808bc1911ff6d51b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e5bbc61e5a97426b808bc1911ff6d51b 2023-04-17 12:58:23,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e5bbc61e5a97426b808bc1911ff6d51b, entries=150, sequenceid=518, filesize=11.9 K 2023-04-17 12:58:23,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/B/82b43f08b00844709ce0a7e999420b43 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/82b43f08b00844709ce0a7e999420b43 2023-04-17 12:58:23,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/82b43f08b00844709ce0a7e999420b43, entries=150, sequenceid=518, filesize=11.9 K 2023-04-17 12:58:23,630 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/.tmp/C/a87188f3280845f4beb85fd14bf06201 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a87188f3280845f4beb85fd14bf06201 2023-04-17 12:58:23,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a87188f3280845f4beb85fd14bf06201, entries=150, sequenceid=518, filesize=11.9 K 2023-04-17 12:58:23,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=0 B/0 for 2a5e7440639effb3958a1a1b6f36fd53 in 1393ms, sequenceid=518, compaction requested=true 2023-04-17 12:58:23,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:23,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:23,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:23,637 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. because compaction request was cancelled 2023-04-17 12:58:23,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. after waiting 0 ms 2023-04-17 12:58:23,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:23,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:23,638 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. because compaction request was cancelled 2023-04-17 12:58:23,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:23,638 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. because compaction request was cancelled 2023-04-17 12:58:23,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:23,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/6387d302bd724768ae1a6be879413e5f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/98a457d4f937441cbd98f5db0315e8ee, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a5aba99e4efc465f984caea969ff8578, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e6585e9d4649468ab485d924d387ba5f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4ac5adc129e04663b2992e5ff0ff99a4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4f7f60eeb1c146738ef9736d6920dd56, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885] to archive 2023-04-17 12:58:23,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:23,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a1ab21b4f2354fe0a81b551c6c7ace1a 2023-04-17 12:58:23,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5151f6107e0b4972912e6115fa5950d8 2023-04-17 12:58:23,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/c7b3aa18e48a4be580c6ffade05a5dfa 2023-04-17 12:58:23,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/3bfa036b61ce4b51ab9f597f7a766461 2023-04-17 12:58:23,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/6387d302bd724768ae1a6be879413e5f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/6387d302bd724768ae1a6be879413e5f 2023-04-17 12:58:23,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/cbebab940c4f43748bcf47d55f1e7081 2023-04-17 12:58:23,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/111f9007005d4272bb82a978d9d69332 2023-04-17 12:58:23,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/7a145c3af83a4e92806401d608d4722f 2023-04-17 12:58:23,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/98a457d4f937441cbd98f5db0315e8ee to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/98a457d4f937441cbd98f5db0315e8ee 2023-04-17 12:58:23,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/feecc5cbadd74a4f9a177fe19a59107d 2023-04-17 12:58:23,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fb27dda1299e4af78b990f81f7cb1d03 2023-04-17 12:58:23,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/733802665b0c486f86f98809c2a61d59 2023-04-17 12:58:23,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a5aba99e4efc465f984caea969ff8578 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/a5aba99e4efc465f984caea969ff8578 2023-04-17 12:58:23,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/55e51087c1124bf0ba407d832c37d9a1 2023-04-17 12:58:23,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/aa216971ed34440f8642e6c15e60ced4 2023-04-17 12:58:23,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/d5809083513c40bca3fc4805bc8bacd2 2023-04-17 12:58:23,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e6585e9d4649468ab485d924d387ba5f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e6585e9d4649468ab485d924d387ba5f 2023-04-17 12:58:23,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/5e36aaaff54347d4ae2a662de6af8479 2023-04-17 12:58:23,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fd7074ff62d04992ad0ec66a2bfc5533 2023-04-17 12:58:23,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4ac5adc129e04663b2992e5ff0ff99a4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4ac5adc129e04663b2992e5ff0ff99a4 2023-04-17 12:58:23,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/baafe55cfb574547b7373660c381d148 2023-04-17 12:58:23,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/1265553e5e1c4b55b521669f787deb8b 2023-04-17 12:58:23,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4f7f60eeb1c146738ef9736d6920dd56 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4f7f60eeb1c146738ef9736d6920dd56 2023-04-17 12:58:23,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/285fa4a3704540028206a16593be1d39 2023-04-17 12:58:23,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/fadeea53cc194c798b6ec2ff6a8db92d 2023-04-17 12:58:23,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/477137be215a464f9140b86721931885 2023-04-17 12:58:23,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/55765e0fbc734cd2bd947e7778722278, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/dfd119cc119048b4967e0156dd389208, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/2dd6ceea22af4f1fa67c4b240086afd0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/c9a70c49d2a343d09b37e04b48c18fde, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/445e8d5f89224eb69ac4b79b218d0a05, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4b651e69058a45eeb7d0174d3fbc784d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105] to archive 2023-04-17 12:58:23,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:23,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/291fb4fa435c4e1f800a224379858e7b 2023-04-17 12:58:23,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d7465319c2094be6bfec92834cc80fba 2023-04-17 12:58:23,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/148e1044bb004d72abdf32b52b9fd078 2023-04-17 12:58:23,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/55765e0fbc734cd2bd947e7778722278 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/55765e0fbc734cd2bd947e7778722278 2023-04-17 12:58:23,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/27ad2d44db3c42e3ad4b3e2e3f724515 2023-04-17 12:58:23,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6662f11951c243978eec8fb0fbb54736 2023-04-17 12:58:23,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/7d702fb5be8c4972a45d03e7ddc7609b 2023-04-17 12:58:23,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d9be0c9f459e4173ba4c05d0f7364483 2023-04-17 12:58:23,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/dfd119cc119048b4967e0156dd389208 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/dfd119cc119048b4967e0156dd389208 2023-04-17 12:58:23,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/b1f692cd7cab4056aef2326412d07560 2023-04-17 12:58:23,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4c117fcead484c1683af43278a3d32b5 2023-04-17 12:58:23,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/0210091b6d164c82afb4878b0cc759f7 2023-04-17 12:58:23,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/2dd6ceea22af4f1fa67c4b240086afd0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/2dd6ceea22af4f1fa67c4b240086afd0 2023-04-17 12:58:23,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/3e8e35453404429ab3059119fd5bf544 2023-04-17 12:58:23,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/6ac0147a09a441bf93a1fc6d375f852e 2023-04-17 12:58:23,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/c9a70c49d2a343d09b37e04b48c18fde to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/c9a70c49d2a343d09b37e04b48c18fde 2023-04-17 12:58:23,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/cba8bc159efa4febae65ebcdec8a0f79 2023-04-17 12:58:23,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e35b5fbe46f54b22978d1536d4fe9714 2023-04-17 12:58:23,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/445e8d5f89224eb69ac4b79b218d0a05 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/445e8d5f89224eb69ac4b79b218d0a05 2023-04-17 12:58:23,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/93fe598cc9d4408c83fb20d11da6f792 2023-04-17 12:58:23,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d8170d7b931a473d99c1a54f761ed294 2023-04-17 12:58:23,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4b651e69058a45eeb7d0174d3fbc784d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/4b651e69058a45eeb7d0174d3fbc784d 2023-04-17 12:58:23,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/51c92b279d874f268e2b0c1eb1e5d6b2 2023-04-17 12:58:23,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/a96517f12c9b47b6bbf4d3a9fb6694bb 2023-04-17 12:58:23,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/ee821da794434a7aa37cfb3caa9e65f7 2023-04-17 12:58:23,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/538e581bee904cf0929bc9a22cffb105 2023-04-17 12:58:23,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/6941f74f696341cdbd5bb8d995517583, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/466aaffac63d459883c0bef39772df6e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e516a11022bd446cb9a98e41ed3d51d5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/d49e200ae016408da2f838f6d94b9ae9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a04069774d6f4761bd07398e90a1abfc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/f68f51c461db438cacaaae9636feb8b8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98] to archive 2023-04-17 12:58:23,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:23,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ac2e6f3f15b346008184a2079405781f 2023-04-17 12:58:23,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8fe7072b43b04fdbbcbce239e657c78f 2023-04-17 12:58:23,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/35b9545b1a254c60a8f707db8f157318 2023-04-17 12:58:23,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/6941f74f696341cdbd5bb8d995517583 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/6941f74f696341cdbd5bb8d995517583 2023-04-17 12:58:23,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/92700b65fc23400196992352d8a8e21e 2023-04-17 12:58:23,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/1646993384714497827bb7250b15d4eb 2023-04-17 12:58:23,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a97d13eb6c754f86a4383cba3ee9ebe4 2023-04-17 12:58:23,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b5ea8030bc4940edbe2debfbf9c0b2e4 2023-04-17 12:58:23,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/466aaffac63d459883c0bef39772df6e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/466aaffac63d459883c0bef39772df6e 2023-04-17 12:58:23,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/81a55181ca6f4bb6b5ca5aadd62b12ef 2023-04-17 12:58:23,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8027158deded4506858bfc3300e5c699 2023-04-17 12:58:23,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e6f8a35f561c42ce9f9d3f3f141c46bc 2023-04-17 12:58:23,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e516a11022bd446cb9a98e41ed3d51d5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e516a11022bd446cb9a98e41ed3d51d5 2023-04-17 12:58:23,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/04aca8ef5d214bc1a7882e0426f9c718 2023-04-17 12:58:23,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/3130b0199f0f43fa9021c3107f3cb7c2 2023-04-17 12:58:23,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/d49e200ae016408da2f838f6d94b9ae9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/d49e200ae016408da2f838f6d94b9ae9 2023-04-17 12:58:23,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/8a4959021a65471188dc81d66a6ca859 2023-04-17 12:58:23,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/dada9bd09c9148b68fcfe0e26a8beabe 2023-04-17 12:58:23,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a04069774d6f4761bd07398e90a1abfc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a04069774d6f4761bd07398e90a1abfc 2023-04-17 12:58:23,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/87a8524f4b8d49fa8b8ecd0038672fc0 2023-04-17 12:58:23,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/9fc2cd6d9464442387cbd68cbce33218 2023-04-17 12:58:23,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/f68f51c461db438cacaaae9636feb8b8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/f68f51c461db438cacaaae9636feb8b8 2023-04-17 12:58:23,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/ad7b824060a846efbf4ad4430a6eae6f 2023-04-17 12:58:23,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a3f718dfac754ec69ebaf21c976b2d79 2023-04-17 12:58:23,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/b04616ade12f4015aa2da2312f4c4ac8 2023-04-17 12:58:23,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/89bfdc947d3642ff8d2c2cd877136f98 2023-04-17 12:58:23,925 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/recovered.edits/521.seqid, newMaxSeqId=521, maxSeqId=1 2023-04-17 12:58:23,927 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53. 2023-04-17 12:58:23,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 2a5e7440639effb3958a1a1b6f36fd53: 2023-04-17 12:58:23,929 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:23,929 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2a5e7440639effb3958a1a1b6f36fd53, regionState=CLOSED 2023-04-17 12:58:23,929 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736303929"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736303929"}]},"ts":"1681736303929"} 2023-04-17 12:58:23,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-17 12:58:23,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 2a5e7440639effb3958a1a1b6f36fd53, server=jenkins-hbase17.apache.org,40215,1681736213280 in 1.5770 sec 2023-04-17 12:58:23,934 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-17 12:58:23,934 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a5e7440639effb3958a1a1b6f36fd53, UNASSIGN in 1.5830 sec 2023-04-17 12:58:23,935 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736303935"}]},"ts":"1681736303935"} 2023-04-17 12:58:23,936 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:58:23,939 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:58:23,942 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6910 sec 2023-04-17 12:58:24,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 12:58:24,366 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-17 12:58:24,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:58:24,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,370 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 12:58:24,371 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,380 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:24,385 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/recovered.edits] 2023-04-17 12:58:24,389 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4cd2c144c5b440d9acbb64f24424b744 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4cd2c144c5b440d9acbb64f24424b744 2023-04-17 12:58:24,390 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4e60ed2915f84bffa76bb7d051482f79 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/4e60ed2915f84bffa76bb7d051482f79 2023-04-17 12:58:24,392 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e5bbc61e5a97426b808bc1911ff6d51b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/A/e5bbc61e5a97426b808bc1911ff6d51b 2023-04-17 12:58:24,422 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/82b43f08b00844709ce0a7e999420b43 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/82b43f08b00844709ce0a7e999420b43 2023-04-17 12:58:24,427 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d74c8b2b2f294d11a90e77da235e5bd9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/d74c8b2b2f294d11a90e77da235e5bd9 2023-04-17 12:58:24,429 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e2ce8f10bd464d51ac36d105ae30aedd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/B/e2ce8f10bd464d51ac36d105ae30aedd 2023-04-17 12:58:24,432 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a87188f3280845f4beb85fd14bf06201 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/a87188f3280845f4beb85fd14bf06201 2023-04-17 12:58:24,433 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e51ee4f8a2d041aa9bb04f0b5bddf694 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/e51ee4f8a2d041aa9bb04f0b5bddf694 2023-04-17 12:58:24,434 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/fa9cf1cb2091499b94d737982c840848 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/C/fa9cf1cb2091499b94d737982c840848 2023-04-17 12:58:24,437 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/recovered.edits/521.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53/recovered.edits/521.seqid 2023-04-17 12:58:24,438 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/2a5e7440639effb3958a1a1b6f36fd53 2023-04-17 12:58:24,438 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:58:24,448 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,455 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:58:24,459 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:58:24,460 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,460 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:58:24,461 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736304461"}]},"ts":"9223372036854775807"} 2023-04-17 12:58:24,463 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:58:24,463 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2a5e7440639effb3958a1a1b6f36fd53, NAME => 'TestAcidGuarantees,,1681736280007.2a5e7440639effb3958a1a1b6f36fd53.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:58:24,464 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:58:24,464 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736304464"}]},"ts":"9223372036854775807"} 2023-04-17 12:58:24,466 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:58:24,468 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,470 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 102 msec 2023-04-17 12:58:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 12:58:24,472 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-17 12:58:24,493 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 226) - Thread LEAK? -, OpenFileDescriptor=552 (was 540) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=867 (was 674) - SystemLoadAverage LEAK? -, ProcessCount=195 (was 195), AvailableMemoryMB=1270 (was 3458) 2023-04-17 12:58:24,511 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=554, MaxFileDescriptor=60000, SystemLoadAverage=867, ProcessCount=195, AvailableMemoryMB=1269 2023-04-17 12:58:24,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:58:24,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:58:24,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:24,523 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:58:24,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-17 12:58:24,524 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:58:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:24,526 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:24,530 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c empty. 2023-04-17 12:58:24,532 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:24,532 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:58:24,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:24,964 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:58:24,996 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => dc27617f350388cfc6da1c8808e4262c, NAME => 'TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:58:25,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:25,418 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:25,419 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing dc27617f350388cfc6da1c8808e4262c, disabling compactions & flushes 2023-04-17 12:58:25,419 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,419 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,419 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. after waiting 0 ms 2023-04-17 12:58:25,419 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,419 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,419 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:25,421 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:58:25,422 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736305422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736305422"}]},"ts":"1681736305422"} 2023-04-17 12:58:25,424 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:58:25,425 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:58:25,425 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736305425"}]},"ts":"1681736305425"} 2023-04-17 12:58:25,426 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:58:25,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, ASSIGN}] 2023-04-17 12:58:25,431 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, ASSIGN 2023-04-17 12:58:25,432 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:58:25,583 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:25,583 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736305583"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736305583"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736305583"}]},"ts":"1681736305583"} 2023-04-17 12:58:25,584 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:25,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:25,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => dc27617f350388cfc6da1c8808e4262c, NAME => 'TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:58:25,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:25,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,741 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,743 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A 2023-04-17 12:58:25,743 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A 2023-04-17 12:58:25,743 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:25,743 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName A 2023-04-17 12:58:25,744 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:25,744 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,750 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B 2023-04-17 12:58:25,750 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B 2023-04-17 12:58:25,751 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:25,751 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName B 2023-04-17 12:58:25,752 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:25,752 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,754 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C 2023-04-17 12:58:25,754 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C 2023-04-17 12:58:25,755 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:25,755 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName C 2023-04-17 12:58:25,756 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:25,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,760 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:58:25,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:25,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:58:25,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened dc27617f350388cfc6da1c8808e4262c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70388436, jitterRate=0.048869431018829346}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:58:25,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:25,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., pid=37, masterSystemTime=1681736305736 2023-04-17 12:58:25,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:25,775 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:25,775 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736305775"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736305775"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736305775"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736305775"}]},"ts":"1681736305775"} 2023-04-17 12:58:25,784 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-17 12:58:25,784 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 in 197 msec 2023-04-17 12:58:25,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-17 12:58:25,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, ASSIGN in 356 msec 2023-04-17 12:58:25,786 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:58:25,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736305786"}]},"ts":"1681736305786"} 2023-04-17 12:58:25,788 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:58:25,806 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:58:25,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2880 sec 2023-04-17 12:58:26,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 12:58:26,638 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-17 12:58:26,639 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x6ba43635 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:26,671 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd9eb59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:26,685 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:26,694 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:26,706 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:58:26,708 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:37486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:58:26,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:58:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:58:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:27,163 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:58:27,170 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:58:27,171 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:58:27,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 12:58:27,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, REOPEN/MOVE}] 2023-04-17 12:58:27,178 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, REOPEN/MOVE 2023-04-17 12:58:27,178 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:27,179 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736307178"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736307178"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736307178"}]},"ts":"1681736307178"} 2023-04-17 12:58:27,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:27,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing dc27617f350388cfc6da1c8808e4262c, disabling compactions & flushes 2023-04-17 12:58:27,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. after waiting 0 ms 2023-04-17 12:58:27,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 12:58:27,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:27,390 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: dc27617f350388cfc6da1c8808e4262c to self. 2023-04-17 12:58:27,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,406 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=CLOSED 2023-04-17 12:58:27,406 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736307406"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736307406"}]},"ts":"1681736307406"} 2023-04-17 12:58:27,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-17 12:58:27,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 in 238 msec 2023-04-17 12:58:27,442 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=true 2023-04-17 12:58:27,596 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:27,596 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736307596"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736307596"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736307596"}]},"ts":"1681736307596"} 2023-04-17 12:58:27,598 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:27,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => dc27617f350388cfc6da1c8808e4262c, NAME => 'TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:58:27,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:27,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,762 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,763 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A 2023-04-17 12:58:27,763 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A 2023-04-17 12:58:27,764 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:27,764 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName A 2023-04-17 12:58:27,767 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:27,768 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,769 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B 2023-04-17 12:58:27,769 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B 2023-04-17 12:58:27,769 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:27,769 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName B 2023-04-17 12:58:27,770 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:27,771 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,775 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C 2023-04-17 12:58:27,775 DEBUG [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C 2023-04-17 12:58:27,776 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:27,776 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region dc27617f350388cfc6da1c8808e4262c columnFamilyName C 2023-04-17 12:58:27,777 INFO [StoreOpener-dc27617f350388cfc6da1c8808e4262c-1] regionserver.HStore(310): Store=dc27617f350388cfc6da1c8808e4262c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:27,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:58:27,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:27,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened dc27617f350388cfc6da1c8808e4262c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60547925, jitterRate=-0.09776560962200165}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:58:27,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:27,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., pid=42, masterSystemTime=1681736307750 2023-04-17 12:58:27,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:27,788 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:27,788 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736307788"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736307788"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736307788"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736307788"}]},"ts":"1681736307788"} 2023-04-17 12:58:27,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-17 12:58:27,792 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 in 191 msec 2023-04-17 12:58:27,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-17 12:58:27,806 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, REOPEN/MOVE in 616 msec 2023-04-17 12:58:27,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-17 12:58:27,809 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 634 msec 2023-04-17 12:58:27,815 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 1.0990 sec 2023-04-17 12:58:27,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-17 12:58:27,816 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x377ba302 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,838 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d243e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,839 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x1eaa86ba to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,859 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a355de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,860 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x1d63a8ce to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,891 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57b6628e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,892 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5d80e5a8 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,933 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da5030e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,934 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x2ea1a2c1 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,949 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74d6bf18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,950 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:27,978 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:27,979 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x52249391 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:28,021 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a9ece3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:28,021 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5ba3ce53 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:28,043 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ded66ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:28,044 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5144fcff to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:28,062 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14d061d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:28,063 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4cc4 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:28,079 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@228ff1a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:28,098 DEBUG [hconnection-0x257aed64-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,106 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,107 DEBUG [hconnection-0x4e9e7d37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,108 DEBUG [hconnection-0x3f43a067-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,109 DEBUG [hconnection-0x33eed0b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,110 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,118 DEBUG [hconnection-0x790acd59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,124 DEBUG [hconnection-0x677e067b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,124 DEBUG [hconnection-0x27adadc3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,127 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,134 DEBUG [hconnection-0x39d0ecef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,136 DEBUG [hconnection-0x3b158fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,136 DEBUG [hconnection-0x2f28ba44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:28,140 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,146 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,146 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,147 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,151 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,151 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33352, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,152 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:33384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:28,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:28,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:28,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:28,168 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:28,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:28,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:28,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:28,169 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:28,169 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:28,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:28,182 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:28,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:28,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:28,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:28,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:28,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:28,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:28,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:28,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:28,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:28,191 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:28,191 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:28,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:28,204 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:28,225 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:28,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736368286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736368288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736368294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736368295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736368296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736368395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736368405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736368405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736368406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736368408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,464 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417dfa2068d72cf496eb75b72334e8c4217 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dfa2068d72cf496eb75b72334e8c4217 2023-04-17 12:58:28,580 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=19, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/9f27958f92f74c1a8441ad433063e72e 2023-04-17 12:58:28,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736368599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736368632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736368632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736368630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736368638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736368919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736368951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736368951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736368961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736368962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,022 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/9307a179b6644bd3849174fec66911d1 2023-04-17 12:58:29,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736369426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:29,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736369467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736369473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736369473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,479 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/c48726a2ad444848b1cc8b8d42671497 2023-04-17 12:58:29,485 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/9f27958f92f74c1a8441ad433063e72e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e 2023-04-17 12:58:29,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:29,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736369482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,489 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e, entries=150, sequenceid=19, filesize=25.1 K 2023-04-17 12:58:29,490 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/9307a179b6644bd3849174fec66911d1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1 2023-04-17 12:58:29,493 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:58:29,519 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1, entries=150, sequenceid=19, filesize=11.6 K 2023-04-17 12:58:29,520 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/c48726a2ad444848b1cc8b8d42671497 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497 2023-04-17 12:58:29,541 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497, entries=150, sequenceid=19, filesize=11.6 K 2023-04-17 12:58:29,550 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for dc27617f350388cfc6da1c8808e4262c in 1346ms, sequenceid=19, compaction requested=false 2023-04-17 12:58:29,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:29,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:29,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:29,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:29,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:29,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:29,574 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,575 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:29,578 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e12816[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:29,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:29,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,647 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,648 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,648 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:29,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:29,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:29,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:29,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:29,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:29,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:29,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:29,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:29,675 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:29,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:29,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:29,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:29,675 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:29,676 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,676 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:29,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,676 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:29,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:29,677 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:29,679 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:29,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:29,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:29,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:29,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:29,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:29,697 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:29,697 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:29,798 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:29,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:29,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:29,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:29,830 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:29,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:29,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:29,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:29,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:29,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:29,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:29,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:29,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:29,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:29,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:29,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:29,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:29,846 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:29,846 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:29,846 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:29,852 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:29,852 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:29,854 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:29,854 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:29,854 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:58:29,854 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:29,855 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:29,855 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:29,855 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:29,855 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:29,855 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:29,967 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177f293dec9d204270bef08dee8d82d16a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177f293dec9d204270bef08dee8d82d16a 2023-04-17 12:58:30,034 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/595e0b4f16da4d57a7e44f2bf62010db 2023-04-17 12:58:30,145 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/af4168c7ab604c02a585839f5c69dc17 2023-04-17 12:58:30,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:30,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:30,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736370533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736370542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736370548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736370549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736370549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,622 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/c82e479d71ef434bb06f59c0e16286ce 2023-04-17 12:58:30,627 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/595e0b4f16da4d57a7e44f2bf62010db as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db 2023-04-17 12:58:30,632 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db, entries=150, sequenceid=37, filesize=25.1 K 2023-04-17 12:58:30,633 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/af4168c7ab604c02a585839f5c69dc17 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17 2023-04-17 12:58:30,647 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17, entries=150, sequenceid=37, filesize=11.6 K 2023-04-17 12:58:30,652 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/c82e479d71ef434bb06f59c0e16286ce as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce 2023-04-17 12:58:30,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736370654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736370662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736370662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736370663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736370663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,683 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce, entries=150, sequenceid=37, filesize=11.6 K 2023-04-17 12:58:30,689 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for dc27617f350388cfc6da1c8808e4262c in 835ms, sequenceid=37, compaction requested=false 2023-04-17 12:58:30,689 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:30,689 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:30,689 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:30,689 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:30,689 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,689 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:30,689 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:30,697 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,697 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:30,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0730b9[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:30,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:30,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:30,750 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:30,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:30,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:30,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:30,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:30,758 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:30,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:30,762 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:30,762 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:30,763 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:30,763 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:30,763 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:30,764 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,765 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,766 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,766 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:30,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,767 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,770 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:30,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:30,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:30,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,771 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,771 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:30,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:30,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:30,772 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:30,772 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:30,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:30,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:30,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:30,872 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:30,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:30,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:30,886 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:30,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:30,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:30,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:30,887 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,887 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,902 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:30,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:30,902 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:30,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:30,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:30,907 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:30,907 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:30,907 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:30,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:30,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:30,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:30,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:30,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:30,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:30,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:30,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:30,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:30,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:30,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:30,938 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,938 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736370912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736370920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736370922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736370926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:30,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736370943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:30,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74c234[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:30,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:30,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,967 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:30,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:30,967 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:30,969 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,969 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:30,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:30,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:30,969 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:30,969 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:30,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:30,972 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:30,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:30,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:30,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:30,972 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:30,973 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,973 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:30,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,973 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:30,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,976 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,979 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:30,979 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:30,980 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:30,982 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:30,982 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:30,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:31,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736371050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736371050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736371051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736371054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736371054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,082 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:31,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:31,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:31,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:31,122 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:31,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:31,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:31,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:31,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:31,141 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:31,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:31,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:31,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:31,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:31,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:31,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:31,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:31,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:31,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:31,171 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:31,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:31,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:31,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:31,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:31,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:31,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,203 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:31,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:31,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:31,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ec79ff[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:31,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:31,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,234 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:31,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:31,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:31,241 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:31,241 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:31,241 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:31,246 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:31,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:31,246 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:31,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736371277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:31,283 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736371282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736371283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:31,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736371284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736371291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,318 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:31,322 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417617e0abff5814ce6b70a64fbf07b5636 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417617e0abff5814ce6b70a64fbf07b5636 2023-04-17 12:58:31,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:31,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:31,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:31,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:31,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:31,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:31,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:31,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:31,362 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:31,363 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,365 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,374 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/7f50bc61827147878b8b61019d269dd9 2023-04-17 12:58:31,465 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:31,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:31,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:31,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/d6e8fb6129c34e0c93404dd606249ee5 2023-04-17 12:58:31,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:31,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:31,487 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:31,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:31,487 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:31,489 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,489 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,493 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:31,493 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:31,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:31,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:31,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,508 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:31,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:31,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:31,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:31,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:31,515 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:31,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:31,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791c556c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:31,518 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:31,518 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,520 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,520 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:31,521 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:31,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:31,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:31,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:31,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/fcbe2ab883a04187b13b9a4072ba7f3b 2023-04-17 12:58:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:31,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:31,581 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:31,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:31,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:31,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:31,581 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:31,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:31,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:31,592 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736371592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:31,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736371595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736371596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,604 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:31,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736371601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:31,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736371602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:31,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/7f50bc61827147878b8b61019d269dd9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9 2023-04-17 12:58:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:31,626 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:31,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,627 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:31,627 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:31,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:31,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:31,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:31,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9, entries=200, sequenceid=53, filesize=31.9 K 2023-04-17 12:58:31,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/d6e8fb6129c34e0c93404dd606249ee5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5 2023-04-17 12:58:31,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5, entries=150, sequenceid=53, filesize=11.6 K 2023-04-17 12:58:31,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/fcbe2ab883a04187b13b9a4072ba7f3b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b 2023-04-17 12:58:31,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b, entries=150, sequenceid=53, filesize=11.6 K 2023-04-17 12:58:31,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for dc27617f350388cfc6da1c8808e4262c in 821ms, sequenceid=53, compaction requested=true 2023-04-17 12:58:31,686 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:31,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:31,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:31,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:58:31,686 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:31,687 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:31,689 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:31,689 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/A is initiating minor compaction (all files) 2023-04-17 12:58:31,689 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/A in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,689 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=82.2 K 2023-04-17 12:58:31,689 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:31,689 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/C is initiating minor compaction (all files) 2023-04-17 12:58:31,689 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/C in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,689 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=34.8 K 2023-04-17 12:58:31,691 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 9f27958f92f74c1a8441ad433063e72e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681736308204 2023-04-17 12:58:31,693 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c48726a2ad444848b1cc8b8d42671497, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681736308204 2023-04-17 12:58:31,693 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 595e0b4f16da4d57a7e44f2bf62010db, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681736308285 2023-04-17 12:58:31,694 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c82e479d71ef434bb06f59c0e16286ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681736308285 2023-04-17 12:58:31,694 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting fcbe2ab883a04187b13b9a4072ba7f3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:31,695 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7f50bc61827147878b8b61019d269dd9, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:31,727 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:31,727 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#C#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:31,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:31,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:31,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:31,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:31,796 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:31,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:31,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:31,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:31,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:31,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:31,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:31,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:31,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:31,828 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#A#compaction#286 average throughput is 0.21 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:31,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:31,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:31,840 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:31,841 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:31,841 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:31,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:31,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:31,842 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,842 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:31,843 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:58:31,843 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:31,843 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:31,843 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:31,844 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:31,844 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/f193a0aa02ff4b799ba5e0c25567e1de as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/f193a0aa02ff4b799ba5e0c25567e1de 2023-04-17 12:58:31,844 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:31,844 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:31,858 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/C of dc27617f350388cfc6da1c8808e4262c into f193a0aa02ff4b799ba5e0c25567e1de(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:31,859 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:31,859 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/C, priority=13, startTime=1681736311686; duration=0sec 2023-04-17 12:58:31,859 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:31,859 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:31,888 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:31,888 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/B is initiating minor compaction (all files) 2023-04-17 12:58:31,888 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/B in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:31,888 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=34.8 K 2023-04-17 12:58:31,892 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9307a179b6644bd3849174fec66911d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681736308204 2023-04-17 12:58:31,896 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting af4168c7ab604c02a585839f5c69dc17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1681736308285 2023-04-17 12:58:31,899 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d6e8fb6129c34e0c93404dd606249ee5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:31,919 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:32,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:32,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:32,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736372138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736372147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736372148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736372150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736372151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736372252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736372283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736372283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736372283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736372283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,419 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417790d5e598daf4dd4b3088238f65a174a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417790d5e598daf4dd4b3088238f65a174a 2023-04-17 12:58:32,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736372464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736372496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736372496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736372496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736372497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,518 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/1bd7bf2dca9448c3b9bd4c507c16f7d6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1bd7bf2dca9448c3b9bd4c507c16f7d6 2023-04-17 12:58:32,521 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/a616be4464dc4bb3a207b9b5427afbe1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a616be4464dc4bb3a207b9b5427afbe1 2023-04-17 12:58:32,567 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/A of dc27617f350388cfc6da1c8808e4262c into a616be4464dc4bb3a207b9b5427afbe1(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:32,567 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:32,567 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/A, priority=13, startTime=1681736311686; duration=0sec 2023-04-17 12:58:32,567 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:32,570 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/B of dc27617f350388cfc6da1c8808e4262c into 1bd7bf2dca9448c3b9bd4c507c16f7d6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:32,570 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:32,570 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/B, priority=13, startTime=1681736311686; duration=0sec 2023-04-17 12:58:32,570 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:32,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736372775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736372809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736372809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736372814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:32,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736372814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:32,890 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/33cd5a1cc0f14268b1e4888773cb477c 2023-04-17 12:58:32,985 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/a4338f138dea43c18afaf2c1d47a2f2b 2023-04-17 12:58:33,064 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7147cc8e8a66495d8d034eb86f8b7c77 2023-04-17 12:58:33,092 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/33cd5a1cc0f14268b1e4888773cb477c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c 2023-04-17 12:58:33,114 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c, entries=150, sequenceid=69, filesize=25.1 K 2023-04-17 12:58:33,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/a4338f138dea43c18afaf2c1d47a2f2b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b 2023-04-17 12:58:33,135 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b, entries=150, sequenceid=69, filesize=11.6 K 2023-04-17 12:58:33,137 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7147cc8e8a66495d8d034eb86f8b7c77 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77 2023-04-17 12:58:33,150 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77, entries=150, sequenceid=69, filesize=11.6 K 2023-04-17 12:58:33,151 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for dc27617f350388cfc6da1c8808e4262c in 1308ms, sequenceid=69, compaction requested=false 2023-04-17 12:58:33,152 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:33,152 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,152 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:33,152 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:33,152 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,156 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:33,156 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:33,163 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,163 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:33,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea9300e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:33,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:33,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,214 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:33,218 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:33,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:33,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:33,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:33,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:33,233 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:33,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:33,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:33,233 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:33,233 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:33,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,240 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,240 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,240 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,241 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:33,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:33,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:33,258 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:33,258 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:33,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:33,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:33,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 12:58:33,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:33,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:33,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:33,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:33,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:33,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:33,362 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:33,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:33,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:33,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736373365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736373379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736373381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:33,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:33,405 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:33,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:33,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:33,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:33,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736373380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736373397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:33,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:33,421 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041741bbdd03a3d0456fb7c89274f4d8cdcb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741bbdd03a3d0456fb7c89274f4d8cdcb 2023-04-17 12:58:33,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:33,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:33,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:33,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:33,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:33,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:33,446 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,446 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:33,446 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:33,446 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,450 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:33,450 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:33,450 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,450 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:33,450 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:33,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23212e8d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:33,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:33,457 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:33,461 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:33,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:33,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:33,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:33,466 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:33,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:33,469 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:33,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:33,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:33,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:33,469 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:33,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,475 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,477 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,482 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:33,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:33,486 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:33,486 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:33,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736373482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736373518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736373519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736373522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736373528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,590 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:33,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:33,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:33,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:33,665 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:33,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:33,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:33,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,671 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:33,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:33,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:33,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:33,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:33,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:33,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:33,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:33,692 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,692 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:33,692 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:33,692 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,694 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:33,694 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:33,694 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,694 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:33,694 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:33,702 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,702 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,706 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736373706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:33,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68c98897[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:33,726 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:33,726 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,731 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,731 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,731 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:33,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:33,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:33,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:33,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736373739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,748 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:33,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:33,750 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:33,750 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:33,750 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:33,750 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:33,750 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:33,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:33,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,750 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:33,751 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,752 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,752 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,752 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,756 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,756 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:33,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:33,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736373744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736373750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736373750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,761 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:33,761 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:33,863 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:33,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:33,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:33,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:33,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:33,878 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:33,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:33,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:33,882 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,882 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,885 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:33,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:33,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,891 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:33,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:33,891 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:33,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:33,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:33,896 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/9b455a0e84a54ddd80468ff7fc502108 2023-04-17 12:58:33,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:33,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:33,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:33,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:33,915 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:33,915 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:33,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:33,916 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:33,916 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:33,917 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,917 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:33,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736517d7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:33,919 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:33,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:33,922 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:33,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:33,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:33,922 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:33,934 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,938 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:33,938 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:33,938 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:33,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:33,961 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:33,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:33,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:33,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:33,961 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:33,971 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,971 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:33,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,971 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:33,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/a3b57755342642d5a13defad1dedbafd 2023-04-17 12:58:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:33,976 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:33,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,983 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:34,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:34,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,007 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,007 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,010 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:34,010 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:34,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736374027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736374062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736374070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736374074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736374074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,110 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:34,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:34,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:34,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:34,122 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:34,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:34,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:34,126 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,126 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,138 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:34,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:34,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:34,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:34,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,154 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:34,161 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:34,161 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:34,162 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,162 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:34,162 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:34,162 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:34,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:34,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:34,166 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:34,178 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:34,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74cb60f5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:34,208 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:34,208 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,211 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:34,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:34,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:34,222 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,222 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:34,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:34,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:34,227 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:34,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:34,274 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:34,274 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:34,274 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:34,274 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:34,274 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:34,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,293 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,306 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:34,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,307 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:34,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,315 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:34,315 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:34,415 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:34,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:34,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:34,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:34,431 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:34,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:34,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:34,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,431 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,434 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,435 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,435 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:34,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:34,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:34,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:34,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:34,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:34,461 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:34,461 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:34,474 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,474 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:34,474 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:34,474 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:34,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:34,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:34,474 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:34,485 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,485 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:34,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb47da7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:34,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:34,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:34,495 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:34,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:34,496 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:34,509 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:34,509 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,509 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:34,509 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/8024a350b0f44f628629c1e0415527cc 2023-04-17 12:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:34,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:34,529 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:34,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:34,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:34,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:34,529 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:34,531 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/9b455a0e84a54ddd80468ff7fc502108 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108 2023-04-17 12:58:34,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,531 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108, entries=150, sequenceid=89, filesize=25.1 K 2023-04-17 12:58:34,535 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/a3b57755342642d5a13defad1dedbafd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd 2023-04-17 12:58:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,549 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd, entries=150, sequenceid=89, filesize=11.6 K 2023-04-17 12:58:34,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:34,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:34,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,550 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:34,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736374542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,550 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:34,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:34,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/8024a350b0f44f628629c1e0415527cc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc 2023-04-17 12:58:34,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736374581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc, entries=150, sequenceid=89, filesize=11.6 K 2023-04-17 12:58:34,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for dc27617f350388cfc6da1c8808e4262c in 1292ms, sequenceid=89, compaction requested=true 2023-04-17 12:58:34,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:34,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:34,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:34,598 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:34,598 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:34,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:34,604 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77341 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:34,604 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/A is initiating minor compaction (all files) 2023-04-17 12:58:34,604 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/A in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,604 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a616be4464dc4bb3a207b9b5427afbe1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=75.5 K 2023-04-17 12:58:34,605 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a616be4464dc4bb3a207b9b5427afbe1, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:34,606 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 33cd5a1cc0f14268b1e4888773cb477c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681736310883 2023-04-17 12:58:34,606 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9b455a0e84a54ddd80468ff7fc502108, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:34,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:58:34,617 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:34,617 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/B is initiating minor compaction (all files) 2023-04-17 12:58:34,617 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/B in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,617 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1bd7bf2dca9448c3b9bd4c507c16f7d6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=34.9 K 2023-04-17 12:58:34,618 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1bd7bf2dca9448c3b9bd4c507c16f7d6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:34,619 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a4338f138dea43c18afaf2c1d47a2f2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681736310883 2023-04-17 12:58:34,620 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a3b57755342642d5a13defad1dedbafd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:34,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:34,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:34,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:34,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:34,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:34,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:34,654 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:34,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:34,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:34,716 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#A#compaction#294 average throughput is 0.26 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:34,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:34,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:34,719 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:34,719 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,719 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:34,719 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:34,721 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#B#compaction#296 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:34,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,746 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736374730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736374738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:34,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736374742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:34,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:34,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:34,781 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:34,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:34,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:34,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:34,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:34,787 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:34,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:34,788 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:34,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:34,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:34,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:34,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736374856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:34,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9dcbd6[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:34,874 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:34,874 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736374862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,882 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/6bbc8c1d72794de88a697cbe3cf3cb21 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6bbc8c1d72794de88a697cbe3cf3cb21 2023-04-17 12:58:34,886 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:34,894 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:34,894 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:34,894 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:34,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:34,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736374878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,897 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,897 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:34,898 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:34,898 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:34,898 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:34,907 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,914 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041781c25d37c8b64f9694f91e2a38284f20 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041781c25d37c8b64f9694f91e2a38284f20 2023-04-17 12:58:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,934 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/B of dc27617f350388cfc6da1c8808e4262c into 6bbc8c1d72794de88a697cbe3cf3cb21(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:34,934 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:34,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:34,934 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/B, priority=13, startTime=1681736314598; duration=0sec 2023-04-17 12:58:34,934 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:34,942 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:34,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,946 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:34,946 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/C is initiating minor compaction (all files) 2023-04-17 12:58:34,946 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/C in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:34,947 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/f193a0aa02ff4b799ba5e0c25567e1de, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=34.9 K 2023-04-17 12:58:34,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,950 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f193a0aa02ff4b799ba5e0c25567e1de, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1681736310537 2023-04-17 12:58:34,954 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7147cc8e8a66495d8d034eb86f8b7c77, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681736310883 2023-04-17 12:58:34,955 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8024a350b0f44f628629c1e0415527cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:34,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:34,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:34,965 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:34,965 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:34,965 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:34,965 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:34,965 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:34,978 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/1494475ee89649eb9e196992602bdf03 2023-04-17 12:58:34,987 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,987 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,987 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:34,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:34,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,010 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#C#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:35,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:35,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:35,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:35,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:35,014 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:35,014 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:35,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:35,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:35,034 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/5aaeb61a9a414c9badfb8fd826ea92b5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5aaeb61a9a414c9badfb8fd826ea92b5 2023-04-17 12:58:35,058 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/C of dc27617f350388cfc6da1c8808e4262c into 5aaeb61a9a414c9badfb8fd826ea92b5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:35,058 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:35,058 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/C, priority=13, startTime=1681736314598; duration=0sec 2023-04-17 12:58:35,059 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:35,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736375084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736375086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,114 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:35,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:35,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736375114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:35,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:35,174 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:35,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:35,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:35,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,175 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,175 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:35,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:35,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:35,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:35,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:35,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,182 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:35,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:35,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:35,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:35,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:35,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:35,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:35,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:35,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:35,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:35,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,210 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,211 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:35,214 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:35,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:35,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@219da0fd[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:35,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:35,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,301 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,301 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:35,301 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:35,301 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:35,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:35,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:35,303 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:35,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:35,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:35,322 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:35,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:35,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:35,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:35,322 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:35,342 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,343 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,344 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,351 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/4bc5cbc215bf40dfa805b49742dc0f1d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/4bc5cbc215bf40dfa805b49742dc0f1d 2023-04-17 12:58:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:35,353 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,378 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/A of dc27617f350388cfc6da1c8808e4262c into 4bc5cbc215bf40dfa805b49742dc0f1d(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:35,378 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:35,378 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/A, priority=13, startTime=1681736314598; duration=0sec 2023-04-17 12:58:35,378 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,387 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,387 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:35,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:35,388 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:35,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:35,390 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:35,390 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:35,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:35,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/cb610e7375ca45a288deca822b5bc26f 2023-04-17 12:58:35,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736375417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736375422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736375436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,490 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:35,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/438bfd0c09db477d8f0d672b08353e20 2023-04-17 12:58:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:35,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:35,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:35,505 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:35,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:35,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:35,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,505 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,514 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,515 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:35,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:35,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:35,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/1494475ee89649eb9e196992602bdf03 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03 2023-04-17 12:58:35,530 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:35,530 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:35,534 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:35,534 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:35,534 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:35,534 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:35,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:35,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:35,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:35,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:35,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:35,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03, entries=150, sequenceid=108, filesize=25.1 K 2023-04-17 12:58:35,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:35,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/cb610e7375ca45a288deca822b5bc26f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f 2023-04-17 12:58:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:35,593 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f, entries=150, sequenceid=108, filesize=11.6 K 2023-04-17 12:58:35,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/438bfd0c09db477d8f0d672b08353e20 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20 2023-04-17 12:58:35,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:35,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736375594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:35,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736375596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20, entries=150, sequenceid=108, filesize=11.6 K 2023-04-17 12:58:35,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for dc27617f350388cfc6da1c8808e4262c in 1024ms, sequenceid=108, compaction requested=false 2023-04-17 12:58:35,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:35,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:35,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7ea0f3[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:35,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:35,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,665 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,666 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:35,666 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:35,666 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:35,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:35,685 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:35,696 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:35,696 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:35,722 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,723 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:35,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:35,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:35,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:35,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:35,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:35,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,738 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:35,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,766 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:35,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:35,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,823 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:35,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:35,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:35,824 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:35,824 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:35,924 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:35,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:35,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:35,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:35,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:35,933 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:35,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:35,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:35,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:35,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:35,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:35,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:35,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:35,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:35,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:35,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:35,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:35,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:35,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:35,936 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:35,936 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:35,936 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:35,937 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:35,937 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:35,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:36,050 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041777691d7c6bfd4a93ae10820861619da1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041777691d7c6bfd4a93ae10820861619da1 2023-04-17 12:58:36,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736376084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736376100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736376099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736376200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736376229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736376229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736376415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736376448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736376448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,502 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/5270f31c7da643d983c7513bb994a18a 2023-04-17 12:58:36,570 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/5b0f066b77bb4060a3eaa91d551ab83d 2023-04-17 12:58:36,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736376754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736376759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:36,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:36,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736376759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,043 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/ef4a0fc174cd4019a3b2e4d289fcaa36 2023-04-17 12:58:37,071 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/5270f31c7da643d983c7513bb994a18a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a 2023-04-17 12:58:37,096 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a, entries=150, sequenceid=124, filesize=25.1 K 2023-04-17 12:58:37,100 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/5b0f066b77bb4060a3eaa91d551ab83d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d 2023-04-17 12:58:37,104 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d, entries=150, sequenceid=124, filesize=11.6 K 2023-04-17 12:58:37,105 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/ef4a0fc174cd4019a3b2e4d289fcaa36 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36 2023-04-17 12:58:37,112 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36, entries=150, sequenceid=124, filesize=11.6 K 2023-04-17 12:58:37,113 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for dc27617f350388cfc6da1c8808e4262c in 1176ms, sequenceid=124, compaction requested=true 2023-04-17 12:58:37,113 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:37,113 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:37,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:37,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:37,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:37,113 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:37,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:37,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73dabdde[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:37,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:37,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,186 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:37,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:37,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:37,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:37,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:37,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:37,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:37,218 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:37,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:37,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:37,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:37,218 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:37,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736316079, max timestamp=1681736317274Num uniques 859; 2023-04-17 12:58:37,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736316079, max timestamp=1681736317274Num uniques 859; 2023-04-17 12:58:37,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736316079, max timestamp=1681736317274Num uniques 859; 2023-04-17 12:58:37,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:37,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:37,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:37,286 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,286 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,286 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,286 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,287 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:37,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:37,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:37,294 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:37,294 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:37,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:37,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-17 12:58:37,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:37,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:37,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:37,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:37,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:37,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:37,398 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:37,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:37,406 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417eaa602fe3c3f46609d4c92f152f42acb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eaa602fe3c3f46609d4c92f152f42acb 2023-04-17 12:58:37,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:37,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:37,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:37,454 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:37,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:37,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:37,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:37,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,479 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:37,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:37,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:37,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:37,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:37,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:37,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:37,482 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:37,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:37,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:37,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:37,482 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:37,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:37,485 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,485 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:37,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736377471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736377475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736317422, max timestamp=1681736317484Num uniques 859; 2023-04-17 12:58:37,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736317422, max timestamp=1681736317484Num uniques 859; 2023-04-17 12:58:37,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736317422, max timestamp=1681736317484Num uniques 859; 2023-04-17 12:58:37,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:37,498 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:37,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@670d1b08[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:37,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:37,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,502 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,502 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:37,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:37,518 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,522 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:37,522 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,522 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:37,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:37,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:37,578 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:37,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:37,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:37,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:37,578 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:37,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:37,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:37,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:37,635 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:37,635 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:37,636 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:37,636 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,636 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:37,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:37,735 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:37,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736377702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:37,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:37,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736377714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736377716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736377718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736377723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:37,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:37,770 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:37,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:37,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:37,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:37,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,772 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,774 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:37,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:37,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:37,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:37,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:37,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:37,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:37,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:37,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:37,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:37,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:37,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:37,796 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:37,796 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:37,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:37,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:37,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:37,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:37,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:37,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:37,822 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:37,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5bd55c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:37,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:37,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736377842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,855 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:37,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:37,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:37,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:37,862 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,863 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736377851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736377851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,870 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/803d7ce8d2e54311838fa8db1ede29ae 2023-04-17 12:58:37,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736377856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:37,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736377862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:37,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:37,886 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:37,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:37,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:37,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:37,886 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:37,894 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,894 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:37,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:37,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,927 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/f2b0ce5fd04649c4844e76c149fafae3 2023-04-17 12:58:37,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:37,958 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:37,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:37,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:37,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:37,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:37,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:37,959 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:37,959 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:37,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,060 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:38,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:38,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736378070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736378070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:38,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736378079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736378079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736378082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:38,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:38,117 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:38,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:38,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:38,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:38,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:38,124 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:38,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:38,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:38,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:38,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:38,125 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,125 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:38,125 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:38,125 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:38,125 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:38,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,126 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,127 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,127 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:38,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:38,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d991fc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:38,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:38,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,135 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:38,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:38,135 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:38,136 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,136 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:38,136 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:38,136 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,137 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:38,137 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:38,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:38,148 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:38,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:38,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:38,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:38,148 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:38,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,153 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,155 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,155 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:38,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:38,161 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:38,161 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:38,261 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:38,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:38,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:38,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:38,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:38,322 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:38,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:38,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:38,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,323 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:38,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:38,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:38,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:38,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:38,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:38,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:38,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:38,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:38,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:38,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:38,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:38,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:38,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:38,334 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:38,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cbf274[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:38,336 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:38,336 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:38,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:38,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:38,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:38,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,341 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:38,341 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:38,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:38,342 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:38,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,347 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,349 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,358 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:38,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:38,361 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:38,362 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:38,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736378390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736378390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736378397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736378398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736378403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/6da8b637164143a0ab79200e6cc61d4c 2023-04-17 12:58:38,466 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:38,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:38,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/803d7ce8d2e54311838fa8db1ede29ae as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae 2023-04-17 12:58:38,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:38,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae, entries=250, sequenceid=150, filesize=38.9 K 2023-04-17 12:58:38,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/f2b0ce5fd04649c4844e76c149fafae3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3 2023-04-17 12:58:38,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:38,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:38,506 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:38,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:38,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:38,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3, entries=150, sequenceid=150, filesize=11.8 K 2023-04-17 12:58:38,509 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,510 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/6da8b637164143a0ab79200e6cc61d4c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c 2023-04-17 12:58:38,511 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,512 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:38,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:38,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:38,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:38,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:38,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:38,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:38,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:38,534 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:38,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:38,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:38,538 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:38,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c, entries=150, sequenceid=150, filesize=11.8 K 2023-04-17 12:58:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:38,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=134.18 KB/137400 for dc27617f350388cfc6da1c8808e4262c in 1253ms, sequenceid=150, compaction requested=true 2023-04-17 12:58:38,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:38,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:38,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:38,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:58:38,555 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,555 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,558 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,578 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:38,578 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/A is initiating minor compaction (all files) 2023-04-17 12:58:38,578 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/A in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,578 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/4bc5cbc215bf40dfa805b49742dc0f1d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=114.6 K 2023-04-17 12:58:38,587 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4bc5cbc215bf40dfa805b49742dc0f1d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:38,587 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:38,587 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/C is initiating minor compaction (all files) 2023-04-17 12:58:38,588 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/C in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,588 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5aaeb61a9a414c9badfb8fd826ea92b5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=46.8 K 2023-04-17 12:58:38,590 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5aaeb61a9a414c9badfb8fd826ea92b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:38,590 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 438bfd0c09db477d8f0d672b08353e20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736313341 2023-04-17 12:58:38,591 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ef4a0fc174cd4019a3b2e4d289fcaa36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1681736314668 2023-04-17 12:58:38,591 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6da8b637164143a0ab79200e6cc61d4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:38,594 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1494475ee89649eb9e196992602bdf03, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736313341 2023-04-17 12:58:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,596 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 5270f31c7da643d983c7513bb994a18a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1681736314668 2023-04-17 12:58:38,596 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 803d7ce8d2e54311838fa8db1ede29ae, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:38,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799162fe[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:38,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:38,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,636 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#C#compaction#306 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:38,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:38,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:38,646 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:38,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:38,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:38,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:38,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:38,667 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:38,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:38,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:38,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:38,667 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:38,689 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#A#compaction#307 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:38,690 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,690 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,690 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:38,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:38,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:38,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,741 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:38,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,742 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:38,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:38,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:38,742 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:38,742 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:38,742 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:38,743 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:38,844 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:38,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:38,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:38,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:38,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:38,857 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:38,857 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,857 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:38,857 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:38,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,872 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:38,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:38,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:38,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:38,895 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,896 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:38,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:38,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:38,903 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:38,903 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:38,903 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:38,909 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:38,909 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:38,910 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:38,910 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:38,910 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:58:38,910 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:38,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:38,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:38,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:38,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:38,911 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:38,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:38,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:38,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736378927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736378934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736378938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736378942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:38,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736378947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,050 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417380a80d23828467e8d1f7a98e4fb24a8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417380a80d23828467e8d1f7a98e4fb24a8 2023-04-17 12:58:39,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736379049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736379071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736379071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736379075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,106 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/291e093f4cd04485a44c22f10f970cfc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/291e093f4cd04485a44c22f10f970cfc 2023-04-17 12:58:39,122 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/50791207689c4cd1ab489403b64ab99a 2023-04-17 12:58:39,146 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/C of dc27617f350388cfc6da1c8808e4262c into 291e093f4cd04485a44c22f10f970cfc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:39,146 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:39,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/C, priority=12, startTime=1681736318554; duration=0sec 2023-04-17 12:58:39,148 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:39,149 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:39,154 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:39,154 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/B is initiating minor compaction (all files) 2023-04-17 12:58:39,154 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/B in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:39,154 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6bbc8c1d72794de88a697cbe3cf3cb21, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=46.8 K 2023-04-17 12:58:39,155 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6bbc8c1d72794de88a697cbe3cf3cb21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1681736312135 2023-04-17 12:58:39,155 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting cb610e7375ca45a288deca822b5bc26f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681736313341 2023-04-17 12:58:39,155 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5b0f066b77bb4060a3eaa91d551ab83d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1681736314668 2023-04-17 12:58:39,156 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f2b0ce5fd04649c4844e76c149fafae3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:39,191 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#B#compaction#310 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:39,196 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/94e61331517c4ca18489d9787f38e167 2023-04-17 12:58:39,266 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/b9b7545d573548a9ad1935da6e0ec5cd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/b9b7545d573548a9ad1935da6e0ec5cd 2023-04-17 12:58:39,283 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/A of dc27617f350388cfc6da1c8808e4262c into b9b7545d573548a9ad1935da6e0ec5cd(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:39,283 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:39,283 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/A, priority=12, startTime=1681736318554; duration=0sec 2023-04-17 12:58:39,284 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:39,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736379269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736379290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736379291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736379290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736379590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736379630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736379630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:39,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736379633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,661 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/2353a387eba74bce8adc934aa0146e33 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/2353a387eba74bce8adc934aa0146e33 2023-04-17 12:58:39,678 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/B of dc27617f350388cfc6da1c8808e4262c into 2353a387eba74bce8adc934aa0146e33(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:39,678 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:39,678 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/B, priority=12, startTime=1681736318554; duration=0sec 2023-04-17 12:58:39,678 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:39,712 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7edee50e2c8b46de892d8961f5cdaf29 2023-04-17 12:58:39,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/50791207689c4cd1ab489403b64ab99a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a 2023-04-17 12:58:39,733 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a, entries=150, sequenceid=187, filesize=25.3 K 2023-04-17 12:58:39,734 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/94e61331517c4ca18489d9787f38e167 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167 2023-04-17 12:58:39,783 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167, entries=150, sequenceid=187, filesize=11.8 K 2023-04-17 12:58:39,795 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7edee50e2c8b46de892d8961f5cdaf29 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29 2023-04-17 12:58:39,845 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29, entries=150, sequenceid=187, filesize=11.8 K 2023-04-17 12:58:39,850 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for dc27617f350388cfc6da1c8808e4262c in 940ms, sequenceid=187, compaction requested=false 2023-04-17 12:58:39,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:39,850 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:39,850 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:39,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:39,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:39,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:39,869 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:39,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:39,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:39,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:39,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c95a9a5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:39,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:39,886 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,896 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,896 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,896 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:39,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:39,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:39,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:39,896 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:39,903 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:39,904 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:39,904 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:39,910 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:39,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:39,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:39,978 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:39,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:39,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:39,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:39,979 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:39,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:39,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:39,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:39,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:39,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,011 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:40,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:40,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:40,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:40,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:40,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:40,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:40,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:40,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,035 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,062 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,062 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,062 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,062 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,063 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,063 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,063 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,063 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:40,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:40,064 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,064 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,064 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:40,064 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:40,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:40,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,149 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d87d5388ed3b4c5fab74fd91dbca5def to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d87d5388ed3b4c5fab74fd91dbca5def 2023-04-17 12:58:40,166 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:40,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:40,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:40,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736320163, max timestamp=1681736320185Num uniques 859; 2023-04-17 12:58:40,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:40,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736320163, max timestamp=1681736320185Num uniques 859; 2023-04-17 12:58:40,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:40,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736320163, max timestamp=1681736320185Num uniques 859; 2023-04-17 12:58:40,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:40,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:40,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:40,206 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:40,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:40,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:40,206 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/f76b34ff75a34fe68f2da0551eb24ff9 2023-04-17 12:58:40,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,214 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:40,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:40,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:40,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:40,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:40,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:40,270 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,270 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:40,270 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:40,270 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:40,270 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:40,272 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,272 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,272 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,272 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:40,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10df9fba[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:40,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:40,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:40,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:40,285 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:40,286 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,286 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:40,290 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:40,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:40,298 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:40,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:40,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:40,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:40,298 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:40,303 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,311 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736380277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736380282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736380282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736380330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736380330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,359 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:40,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:40,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,362 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:40,362 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:40,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736380434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736380435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736380436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,463 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:40,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:40,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736380453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736380454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:40,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:40,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:40,495 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:40,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:40,495 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:40,504 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,504 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,509 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,510 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,510 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,510 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:40,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:40,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:40,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,516 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:40,519 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,519 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:40,519 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:40,519 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:40,519 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:40,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,530 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,538 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:40,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@392e8787[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:40,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:40,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,547 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:40,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:40,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:40,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:40,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:40,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:40,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:40,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:40,578 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:40,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:40,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:40,578 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:40,578 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:40,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:40,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:40,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:40,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,599 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,602 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:40,602 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:40,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33368 deadline: 1681736380647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736380649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736380649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736380670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736380674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,701 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/1d674ba9563049c0815378461152b64e 2023-04-17 12:58:40,704 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:40,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:40,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:40,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:40,742 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:40,742 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,742 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:40,742 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:40,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,745 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,747 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,747 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,748 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,748 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:40,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:40,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:40,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:40,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:40,783 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:40,786 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,786 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:40,786 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:40,786 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:40,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:40,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/38aa399db66d45088ace3c51067cd64c 2023-04-17 12:58:40,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,794 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5a94ce[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:40,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:40,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/f76b34ff75a34fe68f2da0551eb24ff9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9 2023-04-17 12:58:40,814 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,814 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:40,815 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9, entries=150, sequenceid=201, filesize=25.3 K 2023-04-17 12:58:40,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/1d674ba9563049c0815378461152b64e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e 2023-04-17 12:58:40,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:40,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:40,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:40,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e, entries=150, sequenceid=201, filesize=11.8 K 2023-04-17 12:58:40,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/38aa399db66d45088ace3c51067cd64c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c 2023-04-17 12:58:40,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:40,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:40,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c, entries=150, sequenceid=201, filesize=11.8 K 2023-04-17 12:58:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for dc27617f350388cfc6da1c8808e4262c in 838ms, sequenceid=201, compaction requested=true 2023-04-17 12:58:40,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,854 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:40,854 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:40,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:40,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:40,858 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:40,858 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/A is initiating minor compaction (all files) 2023-04-17 12:58:40,858 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:40,858 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/A in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,858 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/b9b7545d573548a9ad1935da6e0ec5cd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=76.2 K 2023-04-17 12:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:40,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:40,860 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:40,861 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:40,861 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:40,861 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:40,861 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:40,886 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b9b7545d573548a9ad1935da6e0ec5cd, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:40,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:40,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,887 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 50791207689c4cd1ab489403b64ab99a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736317706 2023-04-17 12:58:40,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,887 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:40,887 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/B is initiating minor compaction (all files) 2023-04-17 12:58:40,888 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/B in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:40,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,888 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/2353a387eba74bce8adc934aa0146e33, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=35.6 K 2023-04-17 12:58:40,888 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f76b34ff75a34fe68f2da0551eb24ff9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:40,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,888 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2353a387eba74bce8adc934aa0146e33, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:40,890 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 94e61331517c4ca18489d9787f38e167, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736317706 2023-04-17 12:58:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:40,894 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,894 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1d674ba9563049c0815378461152b64e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:40,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,928 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#B#compaction#316 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:40,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:40,947 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:40,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:40,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:40,948 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:40,948 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:40,948 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:40,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:40,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:40,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:40,960 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:40,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:40,961 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:40,991 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#A#compaction#315 average throughput is 0.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:41,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736380984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736380986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736380991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736380991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736381008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,049 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:41,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:41,063 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/ea903d4603d24b58b0b13e457592e8ff as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/ea903d4603d24b58b0b13e457592e8ff 2023-04-17 12:58:41,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:41,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:41,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:41,069 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:41,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:41,069 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:41,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,074 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,074 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:41,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:41,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:41,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:41,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:41,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,085 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:41,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:41,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:41,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:41,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:41,086 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/B of dc27617f350388cfc6da1c8808e4262c into ea903d4603d24b58b0b13e457592e8ff(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:41,086 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:41,086 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/B, priority=13, startTime=1681736320855; duration=0sec 2023-04-17 12:58:41,086 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:41,086 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:41,086 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:41,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:41,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:41,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:41,096 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:41,096 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/C is initiating minor compaction (all files) 2023-04-17 12:58:41,096 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/C in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,097 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/291e093f4cd04485a44c22f10f970cfc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=35.6 K 2023-04-17 12:58:41,097 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 291e093f4cd04485a44c22f10f970cfc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681736316098 2023-04-17 12:58:41,122 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,122 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,142 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7edee50e2c8b46de892d8961f5cdaf29, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681736317706 2023-04-17 12:58:41,143 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 38aa399db66d45088ace3c51067cd64c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:41,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa3d481[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:41,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:41,170 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736381122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736381132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33354 deadline: 1681736381155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,176 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:41,176 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:41,176 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:41,176 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:41,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:41,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:41,181 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,182 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#C#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:41,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:41,205 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:41,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:41,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:41,205 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:41,205 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:41,224 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,224 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,225 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,230 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,248 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,258 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/dce855fb8af44f618bd7457cb293a371 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/dce855fb8af44f618bd7457cb293a371 2023-04-17 12:58:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,276 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,276 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,277 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:41,279 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:41,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:41,279 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:41,279 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:41,290 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/C of dc27617f350388cfc6da1c8808e4262c into dce855fb8af44f618bd7457cb293a371(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:41,290 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:41,290 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/C, priority=13, startTime=1681736320855; duration=0sec 2023-04-17 12:58:41,290 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:41,379 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:41,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:41,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736381374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736381374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736381389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:41,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:41,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:41,406 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:41,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:41,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:41,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,407 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,408 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,408 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:41,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:41,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:41,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:41,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:41,409 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:41,409 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,409 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:41,422 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:41,422 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:41,422 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:41,422 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:41,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:41,423 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,423 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,424 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,424 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:41,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eda420[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:41,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:41,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,426 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,427 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,427 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:41,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:41,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:41,427 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:41,428 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,429 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:41,429 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,429 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:41,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:41,439 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:41,439 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:41,439 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:41,439 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:41,439 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:41,441 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,441 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,442 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,443 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,444 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,449 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,449 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,449 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,449 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:41,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:41,450 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:41,450 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:41,506 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041779eb25b815894e4e9494d2fc81ce0131 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041779eb25b815894e4e9494d2fc81ce0131 2023-04-17 12:58:41,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736381518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33346 deadline: 1681736381522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,550 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:41,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:41,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:41,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:41,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:41,588 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:41,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:41,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:41,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,589 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,590 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:41,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:41,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:41,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:41,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:41,595 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,596 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:41,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:41,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:41,600 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,601 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:41,601 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:41,601 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:41,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:41,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:41,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:41,614 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,614 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,614 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:41,615 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:41,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb219c5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:41,648 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:41,648 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,655 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:41,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:41,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:41,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:41,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:41,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:41,666 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736381694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736381694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:41,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736381702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:41,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:41,714 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:41,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:41,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:41,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:41,714 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:41,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,732 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,734 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:41,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:41,794 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:41,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:41,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:41,795 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:41,795 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:41,895 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:41,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:41,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:41,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:41,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:41,934 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:41,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:41,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:41,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,940 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:41,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:41,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:41,946 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:41,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:41,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:41,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:41,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:41,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:41,950 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:41,950 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,951 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:41,951 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:41,951 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:41,951 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:41,951 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:41,951 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,951 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:41,951 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:41,953 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,953 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:41,959 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:41,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,960 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:41,961 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/0743809326874f6b8884b7d43a4d7116 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/0743809326874f6b8884b7d43a4d7116 2023-04-17 12:58:41,966 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/22414e7b0c5a4875a85e5269cc951c9b 2023-04-17 12:58:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:41,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:41,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54511899[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:41,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:41,979 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,985 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:41,985 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:41,985 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:41,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:41,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:41,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:41,996 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/A of dc27617f350388cfc6da1c8808e4262c into 0743809326874f6b8884b7d43a4d7116(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 12:58:41,996 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:41,996 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/A, priority=13, startTime=1681736320854; duration=1sec 2023-04-17 12:58:41,996 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:42,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:42,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:42,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:42,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:42,061 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:42,061 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:42,061 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:42,061 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:42,061 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:42,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,096 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,207 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,207 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:42,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:42,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:42,207 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:42,207 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:42,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:42,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736382207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736382208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736382219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,307 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:42,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:42,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:42,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:42,323 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:42,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:42,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:42,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,330 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:42,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:42,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:42,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,340 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:42,342 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,346 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:42,346 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:42,349 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:42,350 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:42,350 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:42,350 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:42,350 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:42,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:42,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:42,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:42,354 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:42,358 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,358 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,358 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:42,359 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:42,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a2d94e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:42,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:42,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,370 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:42,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:42,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:42,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:42,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:42,375 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:42,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:42,379 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:42,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:42,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:42,379 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:42,379 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:42,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,399 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,403 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,435 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,435 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,436 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,435 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,436 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:42,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:42,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,437 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:42,437 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:42,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/b9c12b7f70ae43c59a8d70f230d2717d 2023-04-17 12:58:42,541 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:42,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:42,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736382541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:42,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33346 deadline: 1681736382545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:42,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:42,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:42,566 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:42,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:42,566 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:42,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,574 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:42,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:42,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:42,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:42,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:42,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:42,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:42,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7eafce370e8a4af09b6c80ce51419806 2023-04-17 12:58:42,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:42,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:42,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:42,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:42,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:42,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:42,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:42,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:42,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,611 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:42,612 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:42,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b85b5de[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:42,614 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:42,614 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:42,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:42,619 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:42,622 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:42,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:42,626 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:42,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:42,633 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:42,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:42,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:42,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:42,633 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:42,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/22414e7b0c5a4875a85e5269cc951c9b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b 2023-04-17 12:58:42,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,637 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,638 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b, entries=200, sequenceid=238, filesize=32.1 K 2023-04-17 12:58:42,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/b9c12b7f70ae43c59a8d70f230d2717d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d 2023-04-17 12:58:42,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d, entries=150, sequenceid=238, filesize=11.8 K 2023-04-17 12:58:42,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/7eafce370e8a4af09b6c80ce51419806 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806 2023-04-17 12:58:42,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,687 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:42,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:42,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:42,688 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,688 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:42,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:42,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:42,690 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:42,690 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:42,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806, entries=150, sequenceid=238, filesize=11.8 K 2023-04-17 12:58:42,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for dc27617f350388cfc6da1c8808e4262c in 1746ms, sequenceid=238, compaction requested=false 2023-04-17 12:58:42,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:42,794 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:42,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:42,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:42,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:42,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:42,827 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:42,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:42,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:42,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:42,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,843 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:42,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:42,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:42,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:42,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:42,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:42,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:42,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:42,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:42,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:42,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:42,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:42,874 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:58:42,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:42,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:42,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:42,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:42,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:42,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:42,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:42,995 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041739b4240ea377443b9c242ded02615841 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041739b4240ea377443b9c242ded02615841 2023-04-17 12:58:43,067 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/99f06d9a34e44531bbb04d435bed9377 2023-04-17 12:58:43,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:43,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736323263, max timestamp=1681736323294Num uniques 859; 2023-04-17 12:58:43,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:43,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736323263, max timestamp=1681736323294Num uniques 859; 2023-04-17 12:58:43,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:43,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736323263, max timestamp=1681736323294Num uniques 859; 2023-04-17 12:58:43,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:43,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736383374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736323328, max timestamp=1681736323382Num uniques 1018; 2023-04-17 12:58:43,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:58:43,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736323328, max timestamp=1681736323382Num uniques 1018; 2023-04-17 12:58:43,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:58:43,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681736323328, max timestamp=1681736323382Num uniques 1018; 2023-04-17 12:58:43,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 12:58:43,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736383499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736383503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736383510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,549 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/e73240c292654cfcbaee3ed832491c24 2023-04-17 12:58:43,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736383621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736383627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736383627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,642 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/15a288bec64f45c0b8158ac08a1fcfb4 2023-04-17 12:58:43,670 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/99f06d9a34e44531bbb04d435bed9377 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377 2023-04-17 12:58:43,699 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377, entries=150, sequenceid=251, filesize=25.3 K 2023-04-17 12:58:43,701 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/e73240c292654cfcbaee3ed832491c24 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24 2023-04-17 12:58:43,720 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24, entries=150, sequenceid=251, filesize=11.8 K 2023-04-17 12:58:43,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/15a288bec64f45c0b8158ac08a1fcfb4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4 2023-04-17 12:58:43,753 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4, entries=150, sequenceid=251, filesize=11.8 K 2023-04-17 12:58:43,754 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for dc27617f350388cfc6da1c8808e4262c in 880ms, sequenceid=251, compaction requested=true 2023-04-17 12:58:43,754 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:43,754 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:43,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:43,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:43,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:43,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:43,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:43,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:43,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,839 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-17 12:58:43,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:43,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:43,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:43,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:43,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:43,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:43,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:43,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:43,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a2e15a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:43,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:43,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,876 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:43,876 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:43,876 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:43,876 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:43,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:43,890 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:43,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:43,890 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:43,891 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:43,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736383891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736383894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:43,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736383902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:43,954 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:43,954 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:43,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:43,955 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:43,955 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:43,993 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:43,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:43,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:43,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:44,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736384000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,010 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33350 deadline: 1681736384010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:44,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736384023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:44,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:44,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:44,144 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:44,144 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:44,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:44,154 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:44,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:44,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:44,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736384218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736384231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736384242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,250 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:44,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:44,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:44,282 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:44,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:44,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:44,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:44,289 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,289 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:44,299 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:44,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:44,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:44,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:44,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:44,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:44,366 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:44,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:44,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:44,378 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:44,378 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:44,378 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:44,378 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:44,378 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:44,378 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:44,378 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,378 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:44,378 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:44,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:44,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:44,392 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041765f159ab06034bd8a76ab889dcb30bf3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765f159ab06034bd8a76ab889dcb30bf3 2023-04-17 12:58:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6bb607[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:44,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:44,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,446 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:44,446 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:44,447 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:44,450 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/bd9f96fd56d1471084b258a4ec67dae5 2023-04-17 12:58:44,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:44,453 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:44,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:44,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:44,454 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:44,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:44,514 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:44,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:44,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:44,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:44,514 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:44,534 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,538 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,538 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:44,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736384528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,550 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:44,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736384547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736384546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33346 deadline: 1681736384566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:44,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:33384 deadline: 1681736384568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,582 DEBUG [Thread-2048] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4245 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:58:44,590 DEBUG [Thread-2044] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4256 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:58:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:44,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:44,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:44,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:44,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:44,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:44,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:44,691 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:44,691 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:44,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:44,794 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:44,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:44,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:44,818 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:44,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:44,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:44,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:44,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,818 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,819 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:44,819 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:44,819 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,819 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:44,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:44,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:44,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:44,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:44,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:44,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:44,856 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:44,856 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:44,856 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:44,856 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:44,856 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:44,856 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:44,856 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:44,856 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:44,857 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,857 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:44,857 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:44,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,878 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:44,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/fd8f1e8650764d678e9875ff50f1da09 2023-04-17 12:58:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:44,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:44,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a356e6d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:44,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:44,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,948 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:44,948 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:44,948 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:44,948 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:44,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:44,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:44,949 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:44,954 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:44,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:44,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:44,966 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:44,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:44,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:44,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:44,966 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:44,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/195a9b5d27b64d7b9cb93d86f0c9d3b1 2023-04-17 12:58:44,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/bd9f96fd56d1471084b258a4ec67dae5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5 2023-04-17 12:58:44,998 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,002 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,002 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5, entries=200, sequenceid=304, filesize=32.3 K 2023-04-17 12:58:45,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/fd8f1e8650764d678e9875ff50f1da09 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09 2023-04-17 12:58:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,009 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09, entries=150, sequenceid=304, filesize=11.9 K 2023-04-17 12:58:45,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/195a9b5d27b64d7b9cb93d86f0c9d3b1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1 2023-04-17 12:58:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1, entries=150, sequenceid=304, filesize=11.9 K 2023-04-17 12:58:45,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for dc27617f350388cfc6da1c8808e4262c in 1219ms, sequenceid=304, compaction requested=true 2023-04-17 12:58:45,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:45,058 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:45,060 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118208 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:45,060 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/A is initiating minor compaction (all files) 2023-04-17 12:58:45,060 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/A in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,060 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/0743809326874f6b8884b7d43a4d7116, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=115.4 K 2023-04-17 12:58:45,061 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 0743809326874f6b8884b7d43a4d7116, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:45,061 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 22414e7b0c5a4875a85e5269cc951c9b, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681736320216 2023-04-17 12:58:45,062 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 99f06d9a34e44531bbb04d435bed9377, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681736320980 2023-04-17 12:58:45,062 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting bd9f96fd56d1471084b258a4ec67dae5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681736323492 2023-04-17 12:58:45,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:45,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:45,066 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:45,066 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:45,069 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:45,069 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,071 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:45,071 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/B is initiating minor compaction (all files) 2023-04-17 12:58:45,071 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/B in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,071 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/ea903d4603d24b58b0b13e457592e8ff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=47.6 K 2023-04-17 12:58:45,072 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ea903d4603d24b58b0b13e457592e8ff, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:45,072 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b9c12b7f70ae43c59a8d70f230d2717d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681736320216 2023-04-17 12:58:45,072 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e73240c292654cfcbaee3ed832491c24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681736320980 2023-04-17 12:58:45,073 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting fd8f1e8650764d678e9875ff50f1da09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681736323492 2023-04-17 12:58:45,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:45,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:45,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:58:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:45,091 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#B#compaction#328 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:45,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:45,099 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#A#compaction#327 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:45,168 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:45,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:45,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:45,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:45,196 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:45,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:45,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:45,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,206 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,207 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,222 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304176938939726ee4d28af9af1c7bb452533 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176938939726ee4d28af9af1c7bb452533 2023-04-17 12:58:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:45,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:45,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:45,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:45,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:45,231 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,231 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:45,231 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:45,231 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:45,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:45,250 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,250 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,254 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:45,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:45,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf81fb0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:45,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:45,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:45,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:45,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:45,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736385266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736385272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:45,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:45,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736385292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:45,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:45,314 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:45,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:45,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:45,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:45,314 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:45,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,406 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,406 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,410 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:45,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:45,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736385394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736385397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,414 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,418 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:45,419 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:45,419 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:45,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736385419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,521 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:45,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:45,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:45,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:45,580 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:45,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:45,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:45,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,585 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,586 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:45,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:45,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:45,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:45,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:45,599 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:45,600 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,600 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:45,600 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:45,600 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:45,600 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:45,606 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,606 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:45,614 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/81f74a93297c4c22bb981e933866001b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/81f74a93297c4c22bb981e933866001b 2023-04-17 12:58:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:45,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd78544[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:45,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:45,625 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,625 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:45,625 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,625 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:45,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:45,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:45,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:45,627 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,626 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,627 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736385617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736385618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:45,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:45,633 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:45,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:45,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:45,633 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:45,633 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:45,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,636 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,641 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/B of dc27617f350388cfc6da1c8808e4262c into 81f74a93297c4c22bb981e933866001b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:45,641 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:45,641 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/B, priority=12, startTime=1681736325069; duration=0sec 2023-04-17 12:58:45,641 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:45,641 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:45,641 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:45,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:45,642 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:45,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:45,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:45,643 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:45,643 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:45,648 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48738 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:45,648 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): dc27617f350388cfc6da1c8808e4262c/C is initiating minor compaction (all files) 2023-04-17 12:58:45,648 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of dc27617f350388cfc6da1c8808e4262c/C in TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,648 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/dce855fb8af44f618bd7457cb293a371, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp, totalSize=47.6 K 2023-04-17 12:58:45,649 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting dce855fb8af44f618bd7457cb293a371, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681736318938 2023-04-17 12:58:45,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736385639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,654 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7eafce370e8a4af09b6c80ce51419806, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681736320216 2023-04-17 12:58:45,655 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 15a288bec64f45c0b8158ac08a1fcfb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1681736320980 2023-04-17 12:58:45,655 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 195a9b5d27b64d7b9cb93d86f0c9d3b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1681736323492 2023-04-17 12:58:45,664 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/6b072aa2696c465baef21cdd78dd0d2a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/6b072aa2696c465baef21cdd78dd0d2a 2023-04-17 12:58:45,669 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/A of dc27617f350388cfc6da1c8808e4262c into 6b072aa2696c465baef21cdd78dd0d2a(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:45,669 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:45,669 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/A, priority=12, startTime=1681736325058; duration=0sec 2023-04-17 12:58:45,669 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:45,676 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): dc27617f350388cfc6da1c8808e4262c#C#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:45,706 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/f3fc3c6f5eaa4edd9ed3e5936487e055 2023-04-17 12:58:45,736 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/358f249b1aa443dbae392c9080cf4c04 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/358f249b1aa443dbae392c9080cf4c04 2023-04-17 12:58:45,746 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:45,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:45,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:45,768 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in dc27617f350388cfc6da1c8808e4262c/C of dc27617f350388cfc6da1c8808e4262c into 358f249b1aa443dbae392c9080cf4c04(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:45,768 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:45,768 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c., storeName=dc27617f350388cfc6da1c8808e4262c/C, priority=12, startTime=1681736325079; duration=0sec 2023-04-17 12:58:45,769 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:45,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:45,790 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:45,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:45,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:45,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:45,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:45,806 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:45,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:45,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:45,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:45,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:45,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:45,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:45,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:45,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:45,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:45,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:45,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:45,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:45,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:45,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:45,834 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:45,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,869 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,869 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:45,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:45,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9dc3b7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:45,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:45,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,934 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:45,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:45,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:45,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:45,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:45,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736385934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736385934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:45,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736385952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,963 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:45,963 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:45,963 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:45,966 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:46,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:46,002 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:46,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:46,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:46,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:46,002 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:46,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:46,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:46,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:46,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:46,022 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:46,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:46,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:46,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:46,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:46,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,150 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:46,150 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:46,150 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:46,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:46,238 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/338836486dd247638bd669a01ec2457e 2023-04-17 12:58:46,250 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:46,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:46,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/05b2e23113bb4e9ab31d20b074ed3097 2023-04-17 12:58:46,274 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:46,274 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:46,275 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:46,275 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:46,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,279 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:46,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:46,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,281 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:46,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/f3fc3c6f5eaa4edd9ed3e5936487e055 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f3fc3c6f5eaa4edd9ed3e5936487e055 2023-04-17 12:58:46,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:46,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:46,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:46,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:46,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:46,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f3fc3c6f5eaa4edd9ed3e5936487e055, entries=200, sequenceid=316, filesize=32.3 K 2023-04-17 12:58:46,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/338836486dd247638bd669a01ec2457e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/338836486dd247638bd669a01ec2457e 2023-04-17 12:58:46,288 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:46,288 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:46,289 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:46,289 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:46,289 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:46,289 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:46,289 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:46,289 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:46,289 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,289 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:46,289 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:46,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:46,291 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:46,292 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,292 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8a828f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:46,293 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:46,293 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,294 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,294 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:46,294 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:46,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:46,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:46,295 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:46,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:46,296 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:46,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:46,296 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:46,297 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,297 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:46,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,300 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:46,301 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,304 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:46,304 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:46,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:46,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:46,305 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,305 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:46,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:46,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:46,305 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:46,305 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:46,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/338836486dd247638bd669a01ec2457e, entries=150, sequenceid=316, filesize=11.9 K 2023-04-17 12:58:46,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/05b2e23113bb4e9ab31d20b074ed3097 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/05b2e23113bb4e9ab31d20b074ed3097 2023-04-17 12:58:46,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/05b2e23113bb4e9ab31d20b074ed3097, entries=150, sequenceid=316, filesize=11.9 K 2023-04-17 12:58:46,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for dc27617f350388cfc6da1c8808e4262c in 1243ms, sequenceid=316, compaction requested=false 2023-04-17 12:58:46,335 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:46,405 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:46,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:46,420 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:46,420 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:46,420 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:46,420 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:46,420 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,421 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:46,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:46,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,423 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:46,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:46,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:46,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:46,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:46,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:46,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:46,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:46,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:46,430 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:46,430 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. started... 2023-04-17 12:58:46,430 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:46,431 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:46,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. as already flushing 2023-04-17 12:58:46,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:46,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736386530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736386531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736386542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736386656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736386666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736386678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736386878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736386881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:46,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736386891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:46,922 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179f5d451a5f2c40e6baa8c8a0cc457c84 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179f5d451a5f2c40e6baa8c8a0cc457c84 2023-04-17 12:58:47,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736387186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736387190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736387199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,418 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=339, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/370ff1eeadc94c058a912768014683a0 2023-04-17 12:58:47,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33354 deadline: 1681736387694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33350 deadline: 1681736387707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:47,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:33368 deadline: 1681736387709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:47,942 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/06d8150a40ae41708ed000e2ec80eec7 2023-04-17 12:58:48,096 DEBUG [Thread-2055] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52249391 to 127.0.0.1:62647 2023-04-17 12:58:48,096 DEBUG [Thread-2055] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,101 DEBUG [Thread-2059] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5144fcff to 127.0.0.1:62647 2023-04-17 12:58:48,101 DEBUG [Thread-2059] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,102 DEBUG [Thread-2053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:62647 2023-04-17 12:58:48,102 DEBUG [Thread-2053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,102 DEBUG [Thread-2061] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4cc4 to 127.0.0.1:62647 2023-04-17 12:58:48,102 DEBUG [Thread-2061] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,115 DEBUG [Thread-2057] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ba3ce53 to 127.0.0.1:62647 2023-04-17 12:58:48,115 DEBUG [Thread-2057] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,420 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/87f92a0ceda645dbb31207fe079590c0 2023-04-17 12:58:48,438 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/370ff1eeadc94c058a912768014683a0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/370ff1eeadc94c058a912768014683a0 2023-04-17 12:58:48,443 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/370ff1eeadc94c058a912768014683a0, entries=150, sequenceid=339, filesize=25.4 K 2023-04-17 12:58:48,445 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/06d8150a40ae41708ed000e2ec80eec7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/06d8150a40ae41708ed000e2ec80eec7 2023-04-17 12:58:48,450 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/06d8150a40ae41708ed000e2ec80eec7, entries=150, sequenceid=339, filesize=11.9 K 2023-04-17 12:58:48,451 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/87f92a0ceda645dbb31207fe079590c0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/87f92a0ceda645dbb31207fe079590c0 2023-04-17 12:58:48,456 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/87f92a0ceda645dbb31207fe079590c0, entries=150, sequenceid=339, filesize=11.9 K 2023-04-17 12:58:48,457 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for dc27617f350388cfc6da1c8808e4262c in 2027ms, sequenceid=339, compaction requested=true 2023-04-17 12:58:48,457 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:48,457 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:48,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:48,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:48,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:48,457 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:48,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:48,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:48,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,459 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:48,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:48,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75313d1[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:48,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:48,463 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,463 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:48,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:48,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:48,465 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:48,465 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:48,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:48,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:48,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:48,468 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:48,468 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:48,468 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:48,468 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:48,468 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:48,469 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,469 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:48,469 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:48,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:48,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,473 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:48,473 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:48,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:48,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:48,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:48,474 DEBUG [Thread-2052] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:48,474 DEBUG [Thread-2052] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:48,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:48,574 DEBUG [Thread-2052] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:48,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:48,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:48,612 DEBUG [Thread-2044] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eaa86ba to 127.0.0.1:62647 2023-04-17 12:58:48,612 DEBUG [Thread-2044] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,617 DEBUG [Thread-2048] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d80e5a8 to 127.0.0.1:62647 2023-04-17 12:58:48,617 DEBUG [Thread-2048] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,675 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a66d91478ca14f80a95b740e65e41348 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a66d91478ca14f80a95b740e65e41348 2023-04-17 12:58:48,708 DEBUG [Thread-2046] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d63a8ce to 127.0.0.1:62647 2023-04-17 12:58:48,708 DEBUG [Thread-2046] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,721 DEBUG [Thread-2042] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377ba302 to 127.0.0.1:62647 2023-04-17 12:58:48,721 DEBUG [Thread-2042] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,723 DEBUG [Thread-2050] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ea1a2c1 to 127.0.0.1:62647 2023-04-17 12:58:48,723 DEBUG [Thread-2050] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 755 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 2264 rows 2023-04-17 12:58:48,726 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 754 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 2262 rows 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 761 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 2282 rows 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 752 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 2255 rows 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 748 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 2240 rows 2023-04-17 12:58:48,727 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:58:48,727 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ba43635 to 127.0.0.1:62647 2023-04-17 12:58:48,727 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:58:48,728 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:58:48,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:58:48,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:48,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:48,744 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736328744"}]},"ts":"1681736328744"} 2023-04-17 12:58:48,746 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:58:48,831 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:58:48,832 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, UNASSIGN}] 2023-04-17 12:58:48,833 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, UNASSIGN 2023-04-17 12:58:48,837 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:48,837 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736328837"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736328837"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736328837"}]},"ts":"1681736328837"} 2023-04-17 12:58:48,838 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:48,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:48,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing dc27617f350388cfc6da1c8808e4262c, disabling compactions & flushes 2023-04-17 12:58:48,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:49,096 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/94a18ef5bc374035809d83d8e70100d9 2023-04-17 12:58:49,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:49,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/b40fa89bee2444f0944d919365cb39e9 2023-04-17 12:58:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:49,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/fe89e33a418f458e97c3b9e8bb252480 2023-04-17 12:58:49,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/94a18ef5bc374035809d83d8e70100d9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/94a18ef5bc374035809d83d8e70100d9 2023-04-17 12:58:49,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/94a18ef5bc374035809d83d8e70100d9, entries=150, sequenceid=352, filesize=25.4 K 2023-04-17 12:58:49,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/b40fa89bee2444f0944d919365cb39e9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b40fa89bee2444f0944d919365cb39e9 2023-04-17 12:58:49,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b40fa89bee2444f0944d919365cb39e9, entries=150, sequenceid=352, filesize=11.9 K 2023-04-17 12:58:49,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/fe89e33a418f458e97c3b9e8bb252480 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fe89e33a418f458e97c3b9e8bb252480 2023-04-17 12:58:49,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fe89e33a418f458e97c3b9e8bb252480, entries=150, sequenceid=352, filesize=11.9 K 2023-04-17 12:58:49,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for dc27617f350388cfc6da1c8808e4262c in 1367ms, sequenceid=352, compaction requested=true 2023-04-17 12:58:49,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:49,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:49,977 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. because compaction request was cancelled 2023-04-17 12:58:49,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:49,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. after waiting 0 ms 2023-04-17 12:58:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:49,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:49,977 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. because compaction request was cancelled 2023-04-17 12:58:49,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing dc27617f350388cfc6da1c8808e4262c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 12:58:49,977 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. because compaction request was cancelled 2023-04-17 12:58:49,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=A 2023-04-17 12:58:49,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:49,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:49,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=B 2023-04-17 12:58:49,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:49,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK dc27617f350388cfc6da1c8808e4262c, store=C 2023-04-17 12:58:49,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:50,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041741132dfbc625473bb5950d5a713b685d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741132dfbc625473bb5950d5a713b685d 2023-04-17 12:58:50,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/a13f58a87a57454fbb73d7534c76ebb3 2023-04-17 12:58:50,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/6c9c1d1e8799416d800d5f44fa062893 2023-04-17 12:58:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:51,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/5148985d2e794722ad4d470c5116c467 2023-04-17 12:58:51,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/A/a13f58a87a57454fbb73d7534c76ebb3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a13f58a87a57454fbb73d7534c76ebb3 2023-04-17 12:58:51,227 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a13f58a87a57454fbb73d7534c76ebb3, entries=150, sequenceid=359, filesize=25.4 K 2023-04-17 12:58:51,228 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/B/6c9c1d1e8799416d800d5f44fa062893 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6c9c1d1e8799416d800d5f44fa062893 2023-04-17 12:58:51,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6c9c1d1e8799416d800d5f44fa062893, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 12:58:51,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/.tmp/C/5148985d2e794722ad4d470c5116c467 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5148985d2e794722ad4d470c5116c467 2023-04-17 12:58:51,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5148985d2e794722ad4d470c5116c467, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 12:58:51,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for dc27617f350388cfc6da1c8808e4262c in 1260ms, sequenceid=359, compaction requested=true 2023-04-17 12:58:51,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a616be4464dc4bb3a207b9b5427afbe1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/4bc5cbc215bf40dfa805b49742dc0f1d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/b9b7545d573548a9ad1935da6e0ec5cd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/0743809326874f6b8884b7d43a4d7116, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5] to archive 2023-04-17 12:58:51,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:51,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9f27958f92f74c1a8441ad433063e72e 2023-04-17 12:58:51,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/595e0b4f16da4d57a7e44f2bf62010db 2023-04-17 12:58:51,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/7f50bc61827147878b8b61019d269dd9 2023-04-17 12:58:51,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a616be4464dc4bb3a207b9b5427afbe1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a616be4464dc4bb3a207b9b5427afbe1 2023-04-17 12:58:51,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/33cd5a1cc0f14268b1e4888773cb477c 2023-04-17 12:58:51,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/4bc5cbc215bf40dfa805b49742dc0f1d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/4bc5cbc215bf40dfa805b49742dc0f1d 2023-04-17 12:58:51,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/9b455a0e84a54ddd80468ff7fc502108 2023-04-17 12:58:51,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/1494475ee89649eb9e196992602bdf03 2023-04-17 12:58:51,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/5270f31c7da643d983c7513bb994a18a 2023-04-17 12:58:51,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/803d7ce8d2e54311838fa8db1ede29ae 2023-04-17 12:58:51,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/b9b7545d573548a9ad1935da6e0ec5cd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/b9b7545d573548a9ad1935da6e0ec5cd 2023-04-17 12:58:51,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/50791207689c4cd1ab489403b64ab99a 2023-04-17 12:58:51,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/0743809326874f6b8884b7d43a4d7116 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/0743809326874f6b8884b7d43a4d7116 2023-04-17 12:58:51,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f76b34ff75a34fe68f2da0551eb24ff9 2023-04-17 12:58:51,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/22414e7b0c5a4875a85e5269cc951c9b 2023-04-17 12:58:51,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/99f06d9a34e44531bbb04d435bed9377 2023-04-17 12:58:51,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/bd9f96fd56d1471084b258a4ec67dae5 2023-04-17 12:58:51,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1bd7bf2dca9448c3b9bd4c507c16f7d6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6bbc8c1d72794de88a697cbe3cf3cb21, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/2353a387eba74bce8adc934aa0146e33, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/ea903d4603d24b58b0b13e457592e8ff, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09] to archive 2023-04-17 12:58:51,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:51,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/9307a179b6644bd3849174fec66911d1 2023-04-17 12:58:51,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/af4168c7ab604c02a585839f5c69dc17 2023-04-17 12:58:51,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1bd7bf2dca9448c3b9bd4c507c16f7d6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1bd7bf2dca9448c3b9bd4c507c16f7d6 2023-04-17 12:58:51,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/d6e8fb6129c34e0c93404dd606249ee5 2023-04-17 12:58:51,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a4338f138dea43c18afaf2c1d47a2f2b 2023-04-17 12:58:51,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6bbc8c1d72794de88a697cbe3cf3cb21 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6bbc8c1d72794de88a697cbe3cf3cb21 2023-04-17 12:58:51,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/a3b57755342642d5a13defad1dedbafd 2023-04-17 12:58:51,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/cb610e7375ca45a288deca822b5bc26f 2023-04-17 12:58:51,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/5b0f066b77bb4060a3eaa91d551ab83d 2023-04-17 12:58:51,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/2353a387eba74bce8adc934aa0146e33 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/2353a387eba74bce8adc934aa0146e33 2023-04-17 12:58:51,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/f2b0ce5fd04649c4844e76c149fafae3 2023-04-17 12:58:51,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/94e61331517c4ca18489d9787f38e167 2023-04-17 12:58:51,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/ea903d4603d24b58b0b13e457592e8ff to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/ea903d4603d24b58b0b13e457592e8ff 2023-04-17 12:58:51,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/1d674ba9563049c0815378461152b64e 2023-04-17 12:58:51,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b9c12b7f70ae43c59a8d70f230d2717d 2023-04-17 12:58:51,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/e73240c292654cfcbaee3ed832491c24 2023-04-17 12:58:51,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/fd8f1e8650764d678e9875ff50f1da09 2023-04-17 12:58:51,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/f193a0aa02ff4b799ba5e0c25567e1de, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5aaeb61a9a414c9badfb8fd826ea92b5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/291e093f4cd04485a44c22f10f970cfc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/dce855fb8af44f618bd7457cb293a371, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1] to archive 2023-04-17 12:58:51,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:51,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c48726a2ad444848b1cc8b8d42671497 2023-04-17 12:58:51,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/c82e479d71ef434bb06f59c0e16286ce 2023-04-17 12:58:51,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/f193a0aa02ff4b799ba5e0c25567e1de to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/f193a0aa02ff4b799ba5e0c25567e1de 2023-04-17 12:58:51,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fcbe2ab883a04187b13b9a4072ba7f3b 2023-04-17 12:58:51,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7147cc8e8a66495d8d034eb86f8b7c77 2023-04-17 12:58:51,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5aaeb61a9a414c9badfb8fd826ea92b5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5aaeb61a9a414c9badfb8fd826ea92b5 2023-04-17 12:58:51,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/8024a350b0f44f628629c1e0415527cc 2023-04-17 12:58:51,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/438bfd0c09db477d8f0d672b08353e20 2023-04-17 12:58:51,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/ef4a0fc174cd4019a3b2e4d289fcaa36 2023-04-17 12:58:51,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/291e093f4cd04485a44c22f10f970cfc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/291e093f4cd04485a44c22f10f970cfc 2023-04-17 12:58:51,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/6da8b637164143a0ab79200e6cc61d4c 2023-04-17 12:58:51,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7edee50e2c8b46de892d8961f5cdaf29 2023-04-17 12:58:51,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/dce855fb8af44f618bd7457cb293a371 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/dce855fb8af44f618bd7457cb293a371 2023-04-17 12:58:51,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/38aa399db66d45088ace3c51067cd64c 2023-04-17 12:58:51,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/7eafce370e8a4af09b6c80ce51419806 2023-04-17 12:58:51,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/15a288bec64f45c0b8158ac08a1fcfb4 2023-04-17 12:58:51,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/195a9b5d27b64d7b9cb93d86f0c9d3b1 2023-04-17 12:58:51,288 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits/362.seqid, newMaxSeqId=362, maxSeqId=4 2023-04-17 12:58:51,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c. 2023-04-17 12:58:51,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for dc27617f350388cfc6da1c8808e4262c: 2023-04-17 12:58:51,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:51,291 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=dc27617f350388cfc6da1c8808e4262c, regionState=CLOSED 2023-04-17 12:58:51,292 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736331291"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736331291"}]},"ts":"1681736331291"} 2023-04-17 12:58:51,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-17 12:58:51,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure dc27617f350388cfc6da1c8808e4262c, server=jenkins-hbase17.apache.org,40215,1681736213280 in 2.4540 sec 2023-04-17 12:58:51,294 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-17 12:58:51,294 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=dc27617f350388cfc6da1c8808e4262c, UNASSIGN in 2.4610 sec 2023-04-17 12:58:51,295 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736331295"}]},"ts":"1681736331295"} 2023-04-17 12:58:51,296 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:58:51,296 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:58:51,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5680 sec 2023-04-17 12:58:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 12:58:52,849 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-17 12:58:52,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:58:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,853 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 12:58:52,854 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,858 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:52,864 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits] 2023-04-17 12:58:52,870 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/370ff1eeadc94c058a912768014683a0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/370ff1eeadc94c058a912768014683a0 2023-04-17 12:58:52,873 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/6b072aa2696c465baef21cdd78dd0d2a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/6b072aa2696c465baef21cdd78dd0d2a 2023-04-17 12:58:52,874 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/94a18ef5bc374035809d83d8e70100d9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/94a18ef5bc374035809d83d8e70100d9 2023-04-17 12:58:52,876 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a13f58a87a57454fbb73d7534c76ebb3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/a13f58a87a57454fbb73d7534c76ebb3 2023-04-17 12:58:52,877 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f3fc3c6f5eaa4edd9ed3e5936487e055 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/A/f3fc3c6f5eaa4edd9ed3e5936487e055 2023-04-17 12:58:52,880 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/06d8150a40ae41708ed000e2ec80eec7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/06d8150a40ae41708ed000e2ec80eec7 2023-04-17 12:58:52,884 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/338836486dd247638bd669a01ec2457e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/338836486dd247638bd669a01ec2457e 2023-04-17 12:58:52,888 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6c9c1d1e8799416d800d5f44fa062893 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/6c9c1d1e8799416d800d5f44fa062893 2023-04-17 12:58:52,890 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/81f74a93297c4c22bb981e933866001b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/81f74a93297c4c22bb981e933866001b 2023-04-17 12:58:52,891 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b40fa89bee2444f0944d919365cb39e9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/B/b40fa89bee2444f0944d919365cb39e9 2023-04-17 12:58:52,893 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/05b2e23113bb4e9ab31d20b074ed3097 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/05b2e23113bb4e9ab31d20b074ed3097 2023-04-17 12:58:52,895 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/358f249b1aa443dbae392c9080cf4c04 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/358f249b1aa443dbae392c9080cf4c04 2023-04-17 12:58:52,900 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5148985d2e794722ad4d470c5116c467 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/5148985d2e794722ad4d470c5116c467 2023-04-17 12:58:52,902 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/87f92a0ceda645dbb31207fe079590c0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/87f92a0ceda645dbb31207fe079590c0 2023-04-17 12:58:52,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fe89e33a418f458e97c3b9e8bb252480 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/C/fe89e33a418f458e97c3b9e8bb252480 2023-04-17 12:58:52,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits/362.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c/recovered.edits/362.seqid 2023-04-17 12:58:52,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/dc27617f350388cfc6da1c8808e4262c 2023-04-17 12:58:52,910 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:58:52,910 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:58:52,911 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 12:58:52,921 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417380a80d23828467e8d1f7a98e4fb24a8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417380a80d23828467e8d1f7a98e4fb24a8 2023-04-17 12:58:52,930 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041739b4240ea377443b9c242ded02615841 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041739b4240ea377443b9c242ded02615841 2023-04-17 12:58:52,933 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741132dfbc625473bb5950d5a713b685d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741132dfbc625473bb5950d5a713b685d 2023-04-17 12:58:52,935 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741bbdd03a3d0456fb7c89274f4d8cdcb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041741bbdd03a3d0456fb7c89274f4d8cdcb 2023-04-17 12:58:52,937 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417617e0abff5814ce6b70a64fbf07b5636 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417617e0abff5814ce6b70a64fbf07b5636 2023-04-17 12:58:52,938 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765f159ab06034bd8a76ab889dcb30bf3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765f159ab06034bd8a76ab889dcb30bf3 2023-04-17 12:58:52,941 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176938939726ee4d28af9af1c7bb452533 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176938939726ee4d28af9af1c7bb452533 2023-04-17 12:58:52,944 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041777691d7c6bfd4a93ae10820861619da1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041777691d7c6bfd4a93ae10820861619da1 2023-04-17 12:58:52,946 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417790d5e598daf4dd4b3088238f65a174a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417790d5e598daf4dd4b3088238f65a174a 2023-04-17 12:58:52,947 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041779eb25b815894e4e9494d2fc81ce0131 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041779eb25b815894e4e9494d2fc81ce0131 2023-04-17 12:58:52,948 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177f293dec9d204270bef08dee8d82d16a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177f293dec9d204270bef08dee8d82d16a 2023-04-17 12:58:52,950 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041781c25d37c8b64f9694f91e2a38284f20 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041781c25d37c8b64f9694f91e2a38284f20 2023-04-17 12:58:52,951 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179f5d451a5f2c40e6baa8c8a0cc457c84 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179f5d451a5f2c40e6baa8c8a0cc457c84 2023-04-17 12:58:52,952 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a66d91478ca14f80a95b740e65e41348 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a66d91478ca14f80a95b740e65e41348 2023-04-17 12:58:52,954 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d87d5388ed3b4c5fab74fd91dbca5def to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d87d5388ed3b4c5fab74fd91dbca5def 2023-04-17 12:58:52,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 12:58:52,955 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dfa2068d72cf496eb75b72334e8c4217 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417dfa2068d72cf496eb75b72334e8c4217 2023-04-17 12:58:52,959 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eaa602fe3c3f46609d4c92f152f42acb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eaa602fe3c3f46609d4c92f152f42acb 2023-04-17 12:58:52,962 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:58:52,968 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,969 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:58:52,973 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:58:52,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:58:52,975 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736332974"}]},"ts":"9223372036854775807"} 2023-04-17 12:58:52,976 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:58:52,976 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => dc27617f350388cfc6da1c8808e4262c, NAME => 'TestAcidGuarantees,,1681736304518.dc27617f350388cfc6da1c8808e4262c.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:58:52,976 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:58:52,977 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736332976"}]},"ts":"9223372036854775807"} 2023-04-17 12:58:52,978 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:58:52,979 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:52,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 129 msec 2023-04-17 12:58:53,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 12:58:53,156 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-17 12:58:53,166 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=530 (was 554), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=971 (was 867) - SystemLoadAverage LEAK? -, ProcessCount=195 (was 195), AvailableMemoryMB=252 (was 1269) 2023-04-17 12:58:53,180 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=530, MaxFileDescriptor=60000, SystemLoadAverage=971, ProcessCount=195, AvailableMemoryMB=252 2023-04-17 12:58:53,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:58:53,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:58:53,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:58:53,185 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:58:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-17 12:58:53,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:58:53,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 12:58:53,187 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,187 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 empty. 2023-04-17 12:58:53,188 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,188 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:58:53,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 12:58:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 12:58:53,595 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:58:53,597 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 63889a78c730fa94931396cdfa19ec86, NAME => 'TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:58:53,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:53,612 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 63889a78c730fa94931396cdfa19ec86, disabling compactions & flushes 2023-04-17 12:58:53,612 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. after waiting 0 ms 2023-04-17 12:58:53,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,613 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,613 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:53,615 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:58:53,619 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736333618"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736333618"}]},"ts":"1681736333618"} 2023-04-17 12:58:53,620 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:58:53,621 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:58:53,621 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736333621"}]},"ts":"1681736333621"} 2023-04-17 12:58:53,622 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:58:53,624 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, ASSIGN}] 2023-04-17 12:58:53,626 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, ASSIGN 2023-04-17 12:58:53,627 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:58:53,778 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=63889a78c730fa94931396cdfa19ec86, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:53,778 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736333778"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736333778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736333778"}]},"ts":"1681736333778"} 2023-04-17 12:58:53,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:58:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 12:58:53,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 63889a78c730fa94931396cdfa19ec86, NAME => 'TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:58:53,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:58:53,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,936 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,938 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A 2023-04-17 12:58:53,938 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A 2023-04-17 12:58:53,938 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:53,938 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63889a78c730fa94931396cdfa19ec86 columnFamilyName A 2023-04-17 12:58:53,939 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(310): Store=63889a78c730fa94931396cdfa19ec86/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:53,939 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,940 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B 2023-04-17 12:58:53,940 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B 2023-04-17 12:58:53,941 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:53,941 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63889a78c730fa94931396cdfa19ec86 columnFamilyName B 2023-04-17 12:58:53,941 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(310): Store=63889a78c730fa94931396cdfa19ec86/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:53,942 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,943 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C 2023-04-17 12:58:53,943 DEBUG [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C 2023-04-17 12:58:53,943 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:58:53,943 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63889a78c730fa94931396cdfa19ec86 columnFamilyName C 2023-04-17 12:58:53,944 INFO [StoreOpener-63889a78c730fa94931396cdfa19ec86-1] regionserver.HStore(310): Store=63889a78c730fa94931396cdfa19ec86/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:58:53,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:58:53,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:53,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:58:53,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 63889a78c730fa94931396cdfa19ec86; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61987707, jitterRate=-0.07631118595600128}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:58:53,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:53,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., pid=49, masterSystemTime=1681736333931 2023-04-17 12:58:53,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:53,953 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=63889a78c730fa94931396cdfa19ec86, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:53,953 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736333953"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736333953"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736333953"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736333953"}]},"ts":"1681736333953"} 2023-04-17 12:58:53,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-17 12:58:53,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 in 175 msec 2023-04-17 12:58:53,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-17 12:58:53,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, ASSIGN in 331 msec 2023-04-17 12:58:53,957 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:58:53,957 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736333957"}]},"ts":"1681736333957"} 2023-04-17 12:58:53,958 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:58:53,964 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:58:53,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 781 msec 2023-04-17 12:58:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 12:58:54,290 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-17 12:58:54,291 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3e895a5d to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,295 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9e26d67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,297 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,300 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,301 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:58:54,305 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:51326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:58:54,307 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x6ea7ac79 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,315 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a885842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,316 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x51ff7cd4 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,322 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8c6f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,323 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x06846250 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,329 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cb7a788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,330 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x647f95f6 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,336 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41ccedcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,337 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x58ae18e2 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,344 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40e6f624, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,344 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,353 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,353 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x4ea9d51a to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,363 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f4b7807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,363 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x31fd1c1f to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,376 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fc0e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,377 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x209faa3c to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,391 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79db57d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,391 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5345748a to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:58:54,399 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e56ea94, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:58:54,434 DEBUG [hconnection-0x66f9e929-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,438 DEBUG [hconnection-0x3a6d9a73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,440 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,440 DEBUG [hconnection-0x2e8e7be7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:54,445 DEBUG [hconnection-0xc26ad98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,445 DEBUG [hconnection-0x715f62e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,446 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,452 DEBUG [hconnection-0x5e5c8de0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,453 DEBUG [hconnection-0x6680783a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,454 DEBUG [hconnection-0x4d755e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,462 DEBUG [hconnection-0x5da537d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,466 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,465 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,467 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,468 DEBUG [hconnection-0x5d0cb38b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:58:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:54,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:54,469 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:54,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:54,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:54,469 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:54,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:54,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:54,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:54,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,476 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:54,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:54,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:54,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:54,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:54,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:54,487 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:54,486 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,483 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:39154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:58:54,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:54,489 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:54,491 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:54,492 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:54,494 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:54,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:54,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:54,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:54,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736334525, max timestamp=1681736334531Num uniques 859; 2023-04-17 12:58:54,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:54,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736334525, max timestamp=1681736334531Num uniques 859; 2023-04-17 12:58:54,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:54,535 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736334525, max timestamp=1681736334533Num uniques 859; 2023-04-17 12:58:54,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:58:54,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681736334552, max timestamp=1681736334564Num uniques 1018; 2023-04-17 12:58:54,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 12:58:54,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681736334552, max timestamp=1681736334564Num uniques 1018; 2023-04-17 12:58:54,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 12:58:54,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681736334552, max timestamp=1681736334564Num uniques 1018; 2023-04-17 12:58:54,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 12:58:54,586 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/05bf57a02e214a3b9bea61bb116817da 2023-04-17 12:58:54,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681736334577, max timestamp=1681736334609Num uniques 1077; 2023-04-17 12:58:54,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-17 12:58:54,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681736334578, max timestamp=1681736334615Num uniques 1077; 2023-04-17 12:58:54,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 12:58:54,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681736334578, max timestamp=1681736334615Num uniques 1077; 2023-04-17 12:58:54,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 12:58:54,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681736334652, max timestamp=1681736334662Num uniques 1186; 2023-04-17 12:58:54,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=17676, suffix heap size=151312,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=386 2023-04-17 12:58:54,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681736334652, max timestamp=1681736334662Num uniques 1036; 2023-04-17 12:58:54,667 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-17 12:58:54,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681736334652, max timestamp=1681736334662Num uniques 1036; 2023-04-17 12:58:54,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-04-17 12:58:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736394691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736394691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736394696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736394696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736394697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736394804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736394804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736394807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736394807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:54,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:54,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736394808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736395008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736395008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736395014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736395014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736395014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,045 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/3ff87b5a10cb4f448a4f99674b2d5084 2023-04-17 12:58:55,080 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/55f35e0b8a38497e8263271fdef97805 2023-04-17 12:58:55,085 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/05bf57a02e214a3b9bea61bb116817da as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da 2023-04-17 12:58:55,092 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da, entries=150, sequenceid=9, filesize=11.6 K 2023-04-17 12:58:55,093 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/3ff87b5a10cb4f448a4f99674b2d5084 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084 2023-04-17 12:58:55,099 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084, entries=150, sequenceid=9, filesize=11.6 K 2023-04-17 12:58:55,100 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/55f35e0b8a38497e8263271fdef97805 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805 2023-04-17 12:58:55,106 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805, entries=150, sequenceid=9, filesize=11.6 K 2023-04-17 12:58:55,107 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for 63889a78c730fa94931396cdfa19ec86 in 613ms, sequenceid=9, compaction requested=false 2023-04-17 12:58:55,107 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:55,107 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:55,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:55,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:55,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:55,107 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:55,108 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:55,109 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:55,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,110 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:55,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:55,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20895bda[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:55,111 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:55,111 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,112 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:55,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:55,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:55,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,112 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:55,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:55,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:55,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:55,115 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:55,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:55,117 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:55,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:55,117 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:55,118 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:55,118 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:55,119 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,119 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:55,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,119 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:55,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:55,121 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,122 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,125 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,125 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:55,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:55,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:55,126 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:55,126 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:55,226 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:55,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:55,245 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:55,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:55,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:55,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:55,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:55,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681736334679, max timestamp=1681736335319Num uniques 1245; 2023-04-17 12:58:55,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=8930, suffix heap size=157448,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=195 2023-04-17 12:58:55,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=120.76 KB heapSize=330.30 KB 2023-04-17 12:58:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:55,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 12:58:55,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/96a16923ad0e4371b362b8f848dd9a36 2023-04-17 12:58:55,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736395363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736395369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736395370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736395372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736395373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/4ce9b5aaf3284515ab86cb61a5cf0b9e 2023-04-17 12:58:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736395473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736395482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736395482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736395484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736395484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:55,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:55,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,580 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:55,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:55,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:55,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:55,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:55,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:55,586 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:55,587 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:55,587 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:55,587 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:55,587 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:55,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:55,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736395679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736395694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736395695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736395695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736395695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/49f6d1e889b249a08215c6ed5384adb0 2023-04-17 12:58:55,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/96a16923ad0e4371b362b8f848dd9a36 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36 2023-04-17 12:58:55,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36, entries=250, sequenceid=99, filesize=16.2 K 2023-04-17 12:58:55,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/4ce9b5aaf3284515ab86cb61a5cf0b9e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e 2023-04-17 12:58:55,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e, entries=150, sequenceid=99, filesize=11.6 K 2023-04-17 12:58:55,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/49f6d1e889b249a08215c6ed5384adb0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0 2023-04-17 12:58:55,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0, entries=150, sequenceid=99, filesize=11.6 K 2023-04-17 12:58:55,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.17 KB/358576, currentSize=67.09 KB/68700 for 63889a78c730fa94931396cdfa19ec86 in 545ms, sequenceid=99, compaction requested=false 2023-04-17 12:58:55,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:55,932 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:55,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:55,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,942 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:55,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:55,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44edb407[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:55,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:55,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:55,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:55,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:55,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:56,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ed5008967fb648aabb87c41d60bbb733 2023-04-17 12:58:56,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736396022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736396026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736396036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736396038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736396039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,071 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9623a8409f1647bb848fc4eb205d15fd 2023-04-17 12:58:56,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/3dd3b652c1344378868a0a445393dcaf 2023-04-17 12:58:56,113 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:56,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,114 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:56,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ed5008967fb648aabb87c41d60bbb733 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733 2023-04-17 12:58:56,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:56,118 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:56,118 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:56,118 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:56,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733, entries=200, sequenceid=113, filesize=13.9 K 2023-04-17 12:58:56,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9623a8409f1647bb848fc4eb205d15fd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd 2023-04-17 12:58:56,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd, entries=150, sequenceid=113, filesize=11.6 K 2023-04-17 12:58:56,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/3dd3b652c1344378868a0a445393dcaf as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf 2023-04-17 12:58:56,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf, entries=150, sequenceid=113, filesize=11.6 K 2023-04-17 12:58:56,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 63889a78c730fa94931396cdfa19ec86 in 147ms, sequenceid=113, compaction requested=true 2023-04-17 12:58:56,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:56,142 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:56,142 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:56,142 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:56,142 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:56,142 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:56,146 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:56,146 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:58:56,146 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:56,146 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=41.7 K 2023-04-17 12:58:56,146 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:56,146 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:58:56,146 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:56,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=34.8 K 2023-04-17 12:58:56,151 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 05bf57a02e214a3b9bea61bb116817da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681736334492 2023-04-17 12:58:56,151 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3ff87b5a10cb4f448a4f99674b2d5084, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681736334492 2023-04-17 12:58:56,152 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 4ce9b5aaf3284515ab86cb61a5cf0b9e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681736334679 2023-04-17 12:58:56,152 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9623a8409f1647bb848fc4eb205d15fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:56,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 12:58:56,153 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 96a16923ad0e4371b362b8f848dd9a36, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681736334679 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:56,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:56,158 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ed5008967fb648aabb87c41d60bbb733, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:56,166 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#352 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:56,170 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:56,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/692a002123c346e3beb0a5e37fc60438 2023-04-17 12:58:56,196 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/7f433039f1ad40a0b51dee9c1ff6d5ab as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7f433039f1ad40a0b51dee9c1ff6d5ab 2023-04-17 12:58:56,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736396182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736396186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736396192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736396194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,212 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 7f433039f1ad40a0b51dee9c1ff6d5ab(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:56,212 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:56,212 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=13, startTime=1681736336142; duration=0sec 2023-04-17 12:58:56,212 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:56,212 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:58:56,213 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:58:56,214 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:58:56,214 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:56,214 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=34.8 K 2023-04-17 12:58:56,214 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 55f35e0b8a38497e8263271fdef97805, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681736334492 2023-04-17 12:58:56,215 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 49f6d1e889b249a08215c6ed5384adb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681736334679 2023-04-17 12:58:56,215 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3dd3b652c1344378868a0a445393dcaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736396206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,224 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:56,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,257 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:56,257 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:56,257 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:56,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:56,259 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:56,259 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:56,259 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:56,259 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:56,259 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:56,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736396300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736396303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736396307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736396307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/17a0c150ceeb4f5d9afb45fa47a3b05d 2023-04-17 12:58:56,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736396320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/f8803797cf0944e88521ee66ae0f6ee6 2023-04-17 12:58:56,403 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/f0b6029ddc0e4e29960727a2ce19879e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f0b6029ddc0e4e29960727a2ce19879e 2023-04-17 12:58:56,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/692a002123c346e3beb0a5e37fc60438 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438 2023-04-17 12:58:56,494 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into f0b6029ddc0e4e29960727a2ce19879e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:56,494 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:56,494 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=13, startTime=1681736336142; duration=0sec 2023-04-17 12:58:56,494 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:56,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438, entries=200, sequenceid=132, filesize=14.0 K 2023-04-17 12:58:56,513 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,513 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:56,513 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:56,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:56,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,517 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:56,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/17a0c150ceeb4f5d9afb45fa47a3b05d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d 2023-04-17 12:58:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736396510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,520 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:56,520 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:56,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:56,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,521 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:56,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:56,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:56,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:56,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,522 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:56,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:56,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:56,522 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:56,522 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:56,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736396523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736396525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736396525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736396530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d, entries=150, sequenceid=132, filesize=11.7 K 2023-04-17 12:58:56,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/f8803797cf0944e88521ee66ae0f6ee6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6 2023-04-17 12:58:56,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6, entries=150, sequenceid=132, filesize=11.7 K 2023-04-17 12:58:56,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 63889a78c730fa94931396cdfa19ec86 in 421ms, sequenceid=132, compaction requested=false 2023-04-17 12:58:56,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:56,623 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:56,627 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/9dce0b93425d4991823f4642f9b563b1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9dce0b93425d4991823f4642f9b563b1 2023-04-17 12:58:56,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:56,637 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into 9dce0b93425d4991823f4642f9b563b1(size=11.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:56,637 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:56,637 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=13, startTime=1681736336142; duration=0sec 2023-04-17 12:58:56,638 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:56,642 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:56,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:56,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:56,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:56,643 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,643 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:56,643 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:56,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:56,646 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:56,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:56,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,647 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:56,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:56,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:56,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:56,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:56,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:56,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:56,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:56,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:56,658 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:56,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:56,674 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:58:56,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:56,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,675 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:56,675 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,675 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:56,675 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:56,780 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/223daeb8a3374f22b3030d386ac47841 2023-04-17 12:58:56,833 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:56,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:56,859 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9b6dcf2d4d6243119656a1632b4d2349 2023-04-17 12:58:56,862 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:58:56,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736396902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736396902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736396904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736396910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:56,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736396922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:56,946 DEBUG [regionserver/jenkins-hbase17:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-17 12:58:56,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733] to archive 2023-04-17 12:58:56,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:56,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/05bf57a02e214a3b9bea61bb116817da 2023-04-17 12:58:56,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/96a16923ad0e4371b362b8f848dd9a36 2023-04-17 12:58:56,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ed5008967fb648aabb87c41d60bbb733 2023-04-17 12:58:56,987 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd] to archive 2023-04-17 12:58:56,991 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:56,995 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ff87b5a10cb4f448a4f99674b2d5084 2023-04-17 12:58:57,004 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/4ce9b5aaf3284515ab86cb61a5cf0b9e 2023-04-17 12:58:57,006 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9623a8409f1647bb848fc4eb205d15fd 2023-04-17 12:58:57,007 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf] to archive 2023-04-17 12:58:57,009 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:58:57,014 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/55f35e0b8a38497e8263271fdef97805 2023-04-17 12:58:57,015 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/49f6d1e889b249a08215c6ed5384adb0 2023-04-17 12:58:57,017 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase17:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3dd3b652c1344378868a0a445393dcaf 2023-04-17 12:58:57,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736397022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736397023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736397023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736397026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736397046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736397233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736397234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736397235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736397235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736397255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,334 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/7c689077652e4ea5b57f2ab8ce8ad907 2023-04-17 12:58:57,362 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/223daeb8a3374f22b3030d386ac47841 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841 2023-04-17 12:58:57,395 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841, entries=150, sequenceid=148, filesize=11.8 K 2023-04-17 12:58:57,396 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9b6dcf2d4d6243119656a1632b4d2349 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349 2023-04-17 12:58:57,406 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349, entries=150, sequenceid=148, filesize=11.8 K 2023-04-17 12:58:57,407 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/7c689077652e4ea5b57f2ab8ce8ad907 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907 2023-04-17 12:58:57,415 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907, entries=150, sequenceid=148, filesize=11.8 K 2023-04-17 12:58:57,416 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 63889a78c730fa94931396cdfa19ec86 in 742ms, sequenceid=148, compaction requested=true 2023-04-17 12:58:57,416 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:57,416 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:57,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:57,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:57,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:57,426 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,426 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:57,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b7cd8ef[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:57,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:57,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,458 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:57,462 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:57,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:57,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:57,463 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:57,463 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,463 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:57,464 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:57,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:57,467 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:57,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:57,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:57,467 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:57,467 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:57,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,468 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,473 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:57,481 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:57,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:57,482 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:57,482 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:57,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:57,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:57,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/a184af147b0c4dae91b9640c020c5e64 2023-04-17 12:58:57,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736397560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736397564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736397572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,582 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:57,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:57,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736397573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736397575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:57,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:57,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:57,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:57,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:57,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:57,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,607 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:57,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:57,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:57,617 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:57,617 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:57,617 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:57,617 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:57,617 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:57,617 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,617 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:57,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/28aa234676824e80a66ced10d9de9a33 2023-04-17 12:58:57,618 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:57,618 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:57,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:57,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:57,618 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:57,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,619 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,621 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,621 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:57,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ae87ca[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:57,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:57,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,623 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:57,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:57,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:57,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:57,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:57,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:57,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:57,627 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:57,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:57,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:57,627 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:57,627 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:57,629 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,629 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,629 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,630 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,632 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,635 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:57,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:57,635 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:57,635 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:57,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:57,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/47bd0d781e47406fa88a6866f6b9ebfa 2023-04-17 12:58:57,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/a184af147b0c4dae91b9640c020c5e64 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64 2023-04-17 12:58:57,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64, entries=200, sequenceid=168, filesize=14.1 K 2023-04-17 12:58:57,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/28aa234676824e80a66ced10d9de9a33 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33 2023-04-17 12:58:57,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 12:58:57,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/47bd0d781e47406fa88a6866f6b9ebfa as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa 2023-04-17 12:58:57,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736397676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736397681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736397682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 12:58:57,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 63889a78c730fa94931396cdfa19ec86 in 153ms, sequenceid=168, compaction requested=true 2023-04-17 12:58:57,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:57,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:57,700 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:57,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:57,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:58:57,700 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:57,703 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52777 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:57,703 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:58:57,703 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,703 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9dce0b93425d4991823f4642f9b563b1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=51.5 K 2023-04-17 12:58:57,705 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47997 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:57,706 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:58:57,706 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,706 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7f433039f1ad40a0b51dee9c1ff6d5ab, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=46.9 K 2023-04-17 12:58:57,706 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 9dce0b93425d4991823f4642f9b563b1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:57,707 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7f433039f1ad40a0b51dee9c1ff6d5ab, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:57,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:57,707 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:58:57,707 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 692a002123c346e3beb0a5e37fc60438, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681736336019 2023-04-17 12:58:57,707 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 17a0c150ceeb4f5d9afb45fa47a3b05d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681736336019 2023-04-17 12:58:57,708 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 223daeb8a3374f22b3030d386ac47841, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681736336186 2023-04-17 12:58:57,708 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9b6dcf2d4d6243119656a1632b4d2349, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681736336186 2023-04-17 12:58:57,708 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a184af147b0c4dae91b9640c020c5e64, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336896 2023-04-17 12:58:57,708 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 28aa234676824e80a66ced10d9de9a33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336908 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:57,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:57,728 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#363 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:57,736 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:57,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:57,739 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#364 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:57,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:57,774 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:57,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:57,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:57,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:57,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,789 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:57,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:57,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:57,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:57,807 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:57,807 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:57,807 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:57,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:57,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:57,816 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/670fef4af75a403cb884ae240fe89033 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/670fef4af75a403cb884ae240fe89033 2023-04-17 12:58:57,819 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,819 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:57,819 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:57,819 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:57,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:57,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:57,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:57,821 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,821 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,821 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 670fef4af75a403cb884ae240fe89033(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:57,821 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:57,821 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=12, startTime=1681736337700; duration=0sec 2023-04-17 12:58:57,821 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:58:57,821 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:58:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,823 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47997 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:58:57,823 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:58:57,823 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:57,823 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f0b6029ddc0e4e29960727a2ce19879e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=46.9 K 2023-04-17 12:58:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,823 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f0b6029ddc0e4e29960727a2ce19879e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1681736335368 2023-04-17 12:58:57,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:57,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e13510e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:57,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:57,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,824 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f8803797cf0944e88521ee66ae0f6ee6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681736336019 2023-04-17 12:58:57,824 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7c689077652e4ea5b57f2ab8ce8ad907, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681736336186 2023-04-17 12:58:57,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,824 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:57,824 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:57,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:57,825 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 47bd0d781e47406fa88a6866f6b9ebfa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336908 2023-04-17 12:58:57,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:57,825 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,825 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:57,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:57,828 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:57,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:57,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:57,828 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:57,828 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:57,829 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,829 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,829 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:57,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:57,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:57,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:57,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:57,834 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:57,834 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:57,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:57,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:57,852 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#366 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:58:57,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736397894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736397895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736397895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736397904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:57,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736397905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,934 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:57,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:57,982 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:57,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:57,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:57,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:57,983 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,983 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,983 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,984 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:57,984 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:57,984 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:57,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:57,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:57,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:57,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:57,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:57,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:57,999 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:57,999 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,000 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:58,005 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:58,005 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:58,006 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,006 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:58,006 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:58,006 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:58,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:58,006 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,007 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:58,007 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:58,008 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:58,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17778747[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:58,027 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:58,027 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,027 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:58,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:58,028 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:58,028 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,028 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:58,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:58,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:58,030 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:58,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:58,034 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:58,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:58,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:58,034 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:58,034 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:58,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736398028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,042 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,042 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,042 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736398034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,045 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:58,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,087 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:58,087 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:58,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,188 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:58,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:58,189 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/804ea78455824f5b96c541dcfb2f866b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/804ea78455824f5b96c541dcfb2f866b 2023-04-17 12:58:58,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:58,200 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into 804ea78455824f5b96c541dcfb2f866b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:58,200 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:58,200 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=12, startTime=1681736337700; duration=0sec 2023-04-17 12:58:58,200 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:58,204 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:58,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:58,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:58,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/a073a3a412114ea2924ef352a148d917 2023-04-17 12:58:58,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,209 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,210 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:58,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:58,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:58,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:58,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736398209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736398214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736398215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:58,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:58,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:58,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:58,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:58,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:58,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:58,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:58,235 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,236 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,236 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:58,240 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:58,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a618ee6[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:58,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:58,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,252 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,252 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:58,252 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:58,252 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:58,253 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,253 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736398245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,255 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:58,255 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,255 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:58,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736398255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/acaa4ad65af94303bb5e6cce0fc5893b 2023-04-17 12:58:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:58,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:58,278 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:58,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:58,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:58,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:58,278 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:58,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,290 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,295 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,331 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:58,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:58,331 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:58,331 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:58,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:58:58,374 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/4d7827e307254aee98a0793d0c6ffd29 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4d7827e307254aee98a0793d0c6ffd29 2023-04-17 12:58:58,380 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into 4d7827e307254aee98a0793d0c6ffd29(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:58:58,380 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:58,380 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=12, startTime=1681736337700; duration=0sec 2023-04-17 12:58:58,380 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:58:58,432 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:58,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:58,447 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:58,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:58,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:58,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:58,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:58,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:58,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:58,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:58,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:58,463 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:58,463 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,463 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:58,463 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:58,464 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:58,464 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:58:58,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:58:58,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:58:58,464 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:58:58,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:58:58,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:58:58,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad1ceb7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:58:58,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:58:58,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,538 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:58:58,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:58:58,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:58:58,546 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:58:58,551 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:58:58,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:58:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736398559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736398579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:58:58,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:58:58,616 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:58:58,616 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:58:58,616 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:58:58,616 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:58:58,616 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:58:58,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,641 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:58:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:58:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:58:58,642 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:58:58,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:58:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:58:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,729 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,729 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:58:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:58:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:58:58,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736398726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736398726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,732 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:58:58,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:58,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:58:58,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:58:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736398726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,733 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:58:58,733 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:58:58,733 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/46e60d38cd954816832ac73b0d0d1eb8 2023-04-17 12:58:58,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/a073a3a412114ea2924ef352a148d917 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917 2023-04-17 12:58:58,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917, entries=200, sequenceid=182, filesize=14.1 K 2023-04-17 12:58:58,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/acaa4ad65af94303bb5e6cce0fc5893b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b 2023-04-17 12:58:58,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b, entries=150, sequenceid=182, filesize=11.8 K 2023-04-17 12:58:58,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/46e60d38cd954816832ac73b0d0d1eb8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8 2023-04-17 12:58:58,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8, entries=150, sequenceid=182, filesize=11.8 K 2023-04-17 12:58:58,834 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:58:58,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 63889a78c730fa94931396cdfa19ec86 in 1127ms, sequenceid=182, compaction requested=false 2023-04-17 12:58:58,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:58:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:58:58,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:58:58,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:58:58,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:58:58,858 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:58:58,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:58:58,858 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:58:58,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,862 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,866 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:58:58,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:58:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:58:58,867 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:58,867 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:58:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:58:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:58:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:58:58,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:58:58,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:58:58,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:58:58,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:58:58,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:58:58,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:58:58,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:58:58,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:58:58,874 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:58:58,874 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:58:58,875 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:58:58,875 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:58:59,078 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:58:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:58:59,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736399160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736399160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736399278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736399278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,362 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/638e7b11135f4ba1b542ec11bdfc24d8 2023-04-17 12:58:59,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736399494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736399495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,546 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/d138ff62bc82488c81faac786be1b722 2023-04-17 12:58:59,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736399739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736399740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736399744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736399808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:58:59,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:58:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736399808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,102 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/c5a086b2aaa044c6af9a5c5de1f277b4 2023-04-17 12:59:00,170 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/638e7b11135f4ba1b542ec11bdfc24d8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8 2023-04-17 12:59:00,231 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8, entries=150, sequenceid=204, filesize=11.8 K 2023-04-17 12:59:00,233 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/d138ff62bc82488c81faac786be1b722 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722 2023-04-17 12:59:00,243 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722, entries=150, sequenceid=204, filesize=11.8 K 2023-04-17 12:59:00,258 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/c5a086b2aaa044c6af9a5c5de1f277b4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4 2023-04-17 12:59:00,325 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4, entries=150, sequenceid=204, filesize=11.8 K 2023-04-17 12:59:00,340 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=87.22 KB/89310 for 63889a78c730fa94931396cdfa19ec86 in 1465ms, sequenceid=204, compaction requested=true 2023-04-17 12:59:00,340 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:00,340 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:00,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:00,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:00,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:00,342 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:00,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:00,350 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:00,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6e77c2[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:00,353 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:00,353 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,354 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,354 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,354 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:00,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:00,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:00,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:00,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,362 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:00,362 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:00,362 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:00,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:00,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:00,363 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:00,366 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:00,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:00,382 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:00,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:00,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:00,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:00,382 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:00,383 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,383 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,384 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,388 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:00,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:00,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:00,389 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:00,389 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:00,383 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:00,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:00,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:00,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736400416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736400416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,489 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:00,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:00,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:00,525 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:00,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:00,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:00,525 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:00,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,530 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736400528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,538 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:00,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:00,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,545 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,545 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:00,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736400538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:00,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:00,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:00,548 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:00,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:00,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:00,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:00,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:00,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:00,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:00,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:00,554 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:00,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:00,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:00,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:00,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:00,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:00,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150882f7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:00,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:00,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,612 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:00,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:00,612 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:00,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:00,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:00,614 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:00,625 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:00,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:00,638 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:00,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:00,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:00,638 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:00,638 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:00,653 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,654 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:00,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,664 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:00,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:00,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:00,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,695 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:00,695 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:00,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736400740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736400758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,795 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:00,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:00,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:00,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:00,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:00,838 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:00,839 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:00,839 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:00,839 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:00,841 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,841 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/4fe5007c5bb748bf97614cc55114d06f 2023-04-17 12:59:00,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:00,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:00,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:00,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:00,861 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,861 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:00,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:00,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:00,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:00,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:00,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:00,870 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:00,871 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:00,871 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:00,871 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:00,871 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:00,871 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:00,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,882 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,882 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,883 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:00,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2328280e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:00,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:00,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,926 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:00,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:00,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:00,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:00,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:00,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,946 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:00,946 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:00,946 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:00,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:00,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:00,990 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:00,990 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:00,990 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:00,990 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:00,990 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:00,992 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,992 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:00,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,992 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:00,997 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:00,998 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,000 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:01,000 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:01,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,001 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,002 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:01,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:01,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:01,002 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:01,002 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:01,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736401053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736401078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,103 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:01,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:01,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:01,120 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:01,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:01,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:01,120 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:01,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,134 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,143 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:01,146 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:01,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:01,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:01,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:01,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:01,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:01,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:01,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:01,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:01,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:01,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:01,202 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:01,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:01,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:01,202 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:01,226 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:01,234 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,241 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,241 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:01,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:01,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3718feaa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:01,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:01,294 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,304 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:01,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:01,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:01,311 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:01,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:01,312 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:01,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,333 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/b03c4f3e05cb4ba88769ddfee8df872d 2023-04-17 12:59:01,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:01,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:01,338 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:01,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:01,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:01,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:01,338 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:01,377 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,377 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:01,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:01,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,409 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,462 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,540 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:01,540 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:01,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:01,546 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:01,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:01,547 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:01,547 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:01,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:01,546 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,548 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:01,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736401569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736401593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,650 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:01,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:01,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:01,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:01,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:01,686 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:01,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:01,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:01,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:01,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,698 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,698 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:01,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,722 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:01,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:01,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:01,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:01,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:01,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:01,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:01,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:01,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:01,768 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:01,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:01,768 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:01,769 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,769 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:01,769 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:01,769 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,769 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:01,769 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:01,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736401765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736401765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,775 DEBUG [Thread-2595] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4213 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:01,791 DEBUG [Thread-2597] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4214 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:01,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:01,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736401784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,803 DEBUG [Thread-2591] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4238 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:01,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:01,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:01,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/95119c7ad871441a92bff1e662167321 2023-04-17 12:59:01,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:01,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/4fe5007c5bb748bf97614cc55114d06f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f 2023-04-17 12:59:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f, entries=250, sequenceid=222, filesize=16.4 K 2023-04-17 12:59:01,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/b03c4f3e05cb4ba88769ddfee8df872d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d 2023-04-17 12:59:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:01,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:01,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a855fec[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:01,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:01,866 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d, entries=150, sequenceid=222, filesize=11.8 K 2023-04-17 12:59:01,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/95119c7ad871441a92bff1e662167321 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321 2023-04-17 12:59:01,886 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:01,886 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:01,886 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:01,886 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:01,895 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,895 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:01,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321, entries=150, sequenceid=222, filesize=11.8 K 2023-04-17 12:59:01,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 63889a78c730fa94931396cdfa19ec86 in 1553ms, sequenceid=222, compaction requested=true 2023-04-17 12:59:01,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:01,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:01,908 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:01,908 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:01,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:01,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:01,918 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:01,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:01,918 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:01,922 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:01,922 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:01,922 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,922 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/804ea78455824f5b96c541dcfb2f866b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=54.2 K 2023-04-17 12:59:01,926 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:01,926 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:01,926 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:01,926 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/670fef4af75a403cb884ae240fe89033, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=47.2 K 2023-04-17 12:59:01,926 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 804ea78455824f5b96c541dcfb2f866b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336908 2023-04-17 12:59:01,930 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 670fef4af75a403cb884ae240fe89033, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336908 2023-04-17 12:59:01,931 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a073a3a412114ea2924ef352a148d917, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681736337572 2023-04-17 12:59:01,931 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting acaa4ad65af94303bb5e6cce0fc5893b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681736337572 2023-04-17 12:59:01,931 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 638e7b11135f4ba1b542ec11bdfc24d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681736337867 2023-04-17 12:59:01,931 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d138ff62bc82488c81faac786be1b722, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681736337867 2023-04-17 12:59:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:01,938 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4fe5007c5bb748bf97614cc55114d06f, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736339153 2023-04-17 12:59:01,938 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b03c4f3e05cb4ba88769ddfee8df872d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736340335 2023-04-17 12:59:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:01,962 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#375 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:01,976 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#376 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:02,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:02,038 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:02,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:02,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:02,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:02,038 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:02,054 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/083e7f48e22b49dfab9666a269e44cd7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/083e7f48e22b49dfab9666a269e44cd7 2023-04-17 12:59:02,058 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,058 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:02,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,061 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,084 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:02,090 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into 083e7f48e22b49dfab9666a269e44cd7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:02,090 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:02,090 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=12, startTime=1681736341908; duration=0sec 2023-04-17 12:59:02,090 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:02,090 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:02,106 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:02,106 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:02,106 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:02,106 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4d7827e307254aee98a0793d0c6ffd29, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=47.2 K 2023-04-17 12:59:02,110 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4d7827e307254aee98a0793d0c6ffd29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736336908 2023-04-17 12:59:02,115 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 46e60d38cd954816832ac73b0d0d1eb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681736337572 2023-04-17 12:59:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:02,118 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c5a086b2aaa044c6af9a5c5de1f277b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681736337867 2023-04-17 12:59:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:02,130 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 95119c7ad871441a92bff1e662167321, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736340335 2023-04-17 12:59:02,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,174 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#377 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:02,201 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:02,202 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,202 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:02,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:02,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:02,202 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:02,202 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:02,210 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:02,210 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:02,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:02,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:02,302 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:02,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:02,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:02,362 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:02,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:02,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:02,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:02,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:02,380 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:02,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:02,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:02,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:02,381 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,381 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:02,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:02,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:02,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:02,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:02,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:02,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:02,390 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:02,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:02,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:02,392 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:02,392 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:02,395 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:02,395 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:02,484 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/f1cf17f15d5049609fd0d21e8ff389ce as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/f1cf17f15d5049609fd0d21e8ff389ce 2023-04-17 12:59:02,542 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into f1cf17f15d5049609fd0d21e8ff389ce(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:02,542 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:02,542 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=12, startTime=1681736341908; duration=0sec 2023-04-17 12:59:02,542 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:02,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:02,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:02,627 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/d786526d078b4677a11c5b6eb98eadfe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/d786526d078b4677a11c5b6eb98eadfe 2023-04-17 12:59:02,637 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into d786526d078b4677a11c5b6eb98eadfe(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:02,637 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:02,637 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=12, startTime=1681736341908; duration=0sec 2023-04-17 12:59:02,637 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:02,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736402660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736402662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736402770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736402774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,852 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/2dd1395a0842402ca3c7dbe5080dc391 2023-04-17 12:59:02,915 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/787fc1f24cac45c29946495f8ecf941f 2023-04-17 12:59:02,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736402986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:02,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736402990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:03,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39052 deadline: 1681736403297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:03,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39068 deadline: 1681736403297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,362 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/b0d55d9e873844e587673c52a73c6048 2023-04-17 12:59:03,373 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/2dd1395a0842402ca3c7dbe5080dc391 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391 2023-04-17 12:59:03,378 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391, entries=150, sequenceid=237, filesize=11.8 K 2023-04-17 12:59:03,378 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/787fc1f24cac45c29946495f8ecf941f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f 2023-04-17 12:59:03,396 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f, entries=150, sequenceid=237, filesize=11.8 K 2023-04-17 12:59:03,397 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/b0d55d9e873844e587673c52a73c6048 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048 2023-04-17 12:59:03,417 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048, entries=150, sequenceid=237, filesize=11.8 K 2023-04-17 12:59:03,419 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 63889a78c730fa94931396cdfa19ec86 in 1027ms, sequenceid=237, compaction requested=false 2023-04-17 12:59:03,419 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:03,419 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:03,419 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:03,419 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:03,419 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,419 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:03,419 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:03,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,431 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,431 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:03,434 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:03,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:03,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:03,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:03,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bcb6571[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:03,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:03,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,495 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:03,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:03,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:03,502 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:03,502 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:03,502 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:03,506 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:03,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:03,507 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:03,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:03,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:03,507 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:03,507 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:03,510 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:03,511 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,511 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:03,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:03,514 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:03,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:03,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:03,550 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,551 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,551 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:03,551 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:03,551 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:03,551 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:03,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:03,651 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:03,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:03,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:03,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:03,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:03,672 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:03,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:03,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:03,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:03,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:03,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:03,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:03,678 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:03,679 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,679 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:03,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:03,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:03,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:03,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:03,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:03,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:03,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:03,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:03,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:03,686 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:03,686 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:03,686 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 12:59:03,686 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:03,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:03,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:03,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:03,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:03,687 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:03,766 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f5b93e28ed49420bb8ac8be0e28b8dc5 2023-04-17 12:59:03,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:03,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:03,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736403954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:03,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736403958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736404067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736404068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,235 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/873a64b28e1f46b998cffa7275835bd3 2023-04-17 12:59:04,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736404271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736404275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736404586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:04,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736404588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,673 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bb1420fbc010451e8d56d36bd8a3af0f 2023-04-17 12:59:04,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f5b93e28ed49420bb8ac8be0e28b8dc5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5 2023-04-17 12:59:04,702 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 12:59:04,703 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/873a64b28e1f46b998cffa7275835bd3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3 2023-04-17 12:59:04,722 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 12:59:04,730 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bb1420fbc010451e8d56d36bd8a3af0f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f 2023-04-17 12:59:04,754 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 12:59:04,762 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 63889a78c730fa94931396cdfa19ec86 in 1076ms, sequenceid=257, compaction requested=true 2023-04-17 12:59:04,762 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:04,762 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:04,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:04,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:04,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:04,762 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:04,774 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:04,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,782 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:04,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:04,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c1405e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:04,814 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:04,814 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:04,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:04,842 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:04,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:04,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:04,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:04,851 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:04,851 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:04,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:04,914 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:04,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:04,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:04,914 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:04,914 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:04,921 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:04,923 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,923 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:04,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,924 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:04,938 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:04,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:04,956 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:04,956 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:04,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:04,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:04,957 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:04,957 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:04,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:04,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,058 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:05,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:05,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:05,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:05,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:05,086 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:05,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:05,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:05,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,097 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,098 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:05,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:05,101 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:05,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:05,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:05,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:05,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:05,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,112 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:05,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:05,123 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:05,124 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,124 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:05,124 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:05,124 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:05,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:05,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:05,124 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:05,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:05,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:05,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395c1a62[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:05,133 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:05,133 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:05,138 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:05,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:05,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:05,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:05,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/6201bb039dbc4ba5b5eb8583c94aa7e6 2023-04-17 12:59:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736405177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736405178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:05,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:05,204 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:05,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:05,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:05,204 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:05,204 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:05,205 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,205 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,205 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,222 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:05,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:05,222 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:05,222 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:05,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,226 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736405286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736405286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,323 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:05,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:05,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:05,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:05,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:05,378 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:05,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:05,378 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:05,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,387 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,387 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:05,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:05,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:05,410 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:05,410 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,410 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:05,414 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:05,414 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:05,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:05,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:05,418 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:05,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:05,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:05,418 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:05,429 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,430 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:05,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f6489a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:05,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:05,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:05,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:05,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:05,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,494 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:05,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:05,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,495 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:05,495 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736405495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736405501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:05,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:05,602 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:05,602 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:05,602 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:05,602 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:05,602 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:05,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,619 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,620 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,627 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:05,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,628 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,628 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:05,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:05,628 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:05,629 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/950b2bb7ee9d42789e1854afd372f2fb 2023-04-17 12:59:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:05,729 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:05,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:05,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:05,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:05,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:05,778 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:05,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:05,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:05,783 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,783 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39140 deadline: 1681736405782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,791 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:05,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:05,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:05,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:05,805 DEBUG [Thread-2595] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8231 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:05,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:05,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:05,819 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:05,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736405804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,829 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:05,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:05,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:05,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:05,834 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:05,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:05,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39124 deadline: 1681736405830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736405806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,851 DEBUG [Thread-2597] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8274 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:05,861 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:05,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:39164 deadline: 1681736405841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:05,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e64838[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:05,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:05,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,864 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:05,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:05,864 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:05,864 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,864 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:05,865 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:05,865 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,865 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,865 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,879 DEBUG [Thread-2591] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8302 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:05,885 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:05,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:05,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:05,885 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:05,885 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:05,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:05,887 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:05,888 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,890 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,893 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:05,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:05,895 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:05,896 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:05,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:05,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:05,896 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:05,896 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:05,996 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:05,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:06,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:06,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:06,026 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:06,026 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,026 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:06,026 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:06,030 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,030 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,034 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:06,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:06,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:06,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:06,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:06,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:06,056 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:06,056 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:06,056 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,056 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:06,057 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:06,057 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:06,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:06,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:06,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:06,063 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,074 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:06,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae628ea[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:06,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:06,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:06,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:06,090 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:06,094 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,094 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:06,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:06,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,098 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:06,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:06,127 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:06,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:06,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:06,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:06,127 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:06,142 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,142 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,154 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,156 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,159 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,160 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,160 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:06,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:06,162 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:06,162 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:06,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/411cedeb688949f0842dfc312286c2a3 2023-04-17 12:59:06,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/6201bb039dbc4ba5b5eb8583c94aa7e6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6 2023-04-17 12:59:06,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6, entries=200, sequenceid=273, filesize=14.3 K 2023-04-17 12:59:06,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/950b2bb7ee9d42789e1854afd372f2fb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb 2023-04-17 12:59:06,262 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:06,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:06,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb, entries=150, sequenceid=273, filesize=11.9 K 2023-04-17 12:59:06,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/411cedeb688949f0842dfc312286c2a3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3 2023-04-17 12:59:06,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:06,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:06,282 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:06,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:06,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:06,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,283 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:06,285 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,285 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,285 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:06,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:06,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:06,286 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:06,286 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,286 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:06,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:06,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:06,290 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:06,290 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:06,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,298 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,299 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,299 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:06,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3, entries=150, sequenceid=273, filesize=11.9 K 2023-04-17 12:59:06,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 63889a78c730fa94931396cdfa19ec86 in 1204ms, sequenceid=273, compaction requested=true 2023-04-17 12:59:06,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:06,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:06,305 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:06,305 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:06,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:06,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:06,309 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:06,309 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:06,309 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,309 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/083e7f48e22b49dfab9666a269e44cd7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=49.8 K 2023-04-17 12:59:06,310 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48568 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:06,310 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,310 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,310 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/f1cf17f15d5049609fd0d21e8ff389ce, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=47.4 K 2023-04-17 12:59:06,310 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 083e7f48e22b49dfab9666a269e44cd7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736340335 2023-04-17 12:59:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,312 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting f1cf17f15d5049609fd0d21e8ff389ce, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736340335 2023-04-17 12:59:06,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:06,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330a6de[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:06,312 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:06,312 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,314 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2dd1395a0842402ca3c7dbe5080dc391, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681736340403 2023-04-17 12:59:06,314 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:06,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:06,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:06,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,315 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:06,316 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 787fc1f24cac45c29946495f8ecf941f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681736340403 2023-04-17 12:59:06,316 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:06,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,316 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,317 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 873a64b28e1f46b998cffa7275835bd3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681736342643 2023-04-17 12:59:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,317 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f5b93e28ed49420bb8ac8be0e28b8dc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681736342643 2023-04-17 12:59:06,318 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 950b2bb7ee9d42789e1854afd372f2fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:06,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:06,319 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:06,319 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:06,319 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:06,319 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:06,320 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:06,326 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6201bb039dbc4ba5b5eb8583c94aa7e6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:06,327 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,327 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,332 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:06,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 12:59:06,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:06,338 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:06,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:06,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:06,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:06,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:06,339 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:06,339 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:06,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,339 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:06,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:06,345 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#387 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:06,352 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#389 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:06,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736406420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,439 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:06,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:06,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,442 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/b47c9b62bc6f4e8292a45861a0a66541 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b47c9b62bc6f4e8292a45861a0a66541 2023-04-17 12:59:06,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736406428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:06,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:06,461 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:06,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:06,461 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:06,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,466 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,468 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into b47c9b62bc6f4e8292a45861a0a66541(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:06,468 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:06,468 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=12, startTime=1681736346305; duration=0sec 2023-04-17 12:59:06,468 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:06,468 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:06,470 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,471 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48568 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:06,471 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:06,471 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,471 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/d786526d078b4677a11c5b6eb98eadfe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=47.4 K 2023-04-17 12:59:06,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,473 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d786526d078b4677a11c5b6eb98eadfe, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681736340335 2023-04-17 12:59:06,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,478 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:06,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:06,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:06,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:06,481 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b0d55d9e873844e587673c52a73c6048, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681736340403 2023-04-17 12:59:06,485 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting bb1420fbc010451e8d56d36bd8a3af0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681736342643 2023-04-17 12:59:06,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:06,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,486 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:06,490 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 411cedeb688949f0842dfc312286c2a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:06,498 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:06,498 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:06,498 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:06,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:06,499 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:06,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:06,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:06,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:06,514 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,518 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736406529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:06,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58598cba[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:06,534 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:06,534 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:06,542 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:06,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:06,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:06,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736406548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:06,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:06,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:06,576 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:06,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:06,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:06,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:06,576 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:06,578 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,579 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,580 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,581 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,584 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#390 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:06,590 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,590 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,590 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,591 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:06,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:06,594 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:06,594 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:06,594 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,594 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,698 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:06,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:06,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:06,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:06,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:06,714 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:06,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:06,714 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:06,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,726 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,726 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,734 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,734 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736406734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:06,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:06,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:06,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:06,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:06,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,754 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:06,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:06,770 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,770 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:06,770 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:06,770 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:06,770 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:06,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:06,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:06,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:06,774 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:06,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:06,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736406776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,792 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:06,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:06,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7355b801[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:06,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:06,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:06,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:06,807 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:06,808 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,809 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:06,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/81bac9a92cd94ca48afb7ac981e804db 2023-04-17 12:59:06,821 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:06,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,827 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/b1eebb18ae5a4342a8c55843f33b8c9d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/b1eebb18ae5a4342a8c55843f33b8c9d 2023-04-17 12:59:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:06,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:06,827 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:06,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:06,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:06,827 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:06,827 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:06,828 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:06,829 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,831 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:06,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:06,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:06,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:06,835 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:06,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:06,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:06,836 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:06,836 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:06,836 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into b1eebb18ae5a4342a8c55843f33b8c9d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:06,836 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:06,836 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=12, startTime=1681736346305; duration=0sec 2023-04-17 12:59:06,837 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:06,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/454bed70e4a8479fbb0a6d5dbe50d43c 2023-04-17 12:59:06,938 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:06,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:06,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/9da8caffc2c34eb3a4686e3afc23e532 2023-04-17 12:59:06,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/81bac9a92cd94ca48afb7ac981e804db as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db 2023-04-17 12:59:06,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:06,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db, entries=200, sequenceid=290, filesize=14.3 K 2023-04-17 12:59:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:06,966 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:06,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/454bed70e4a8479fbb0a6d5dbe50d43c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c 2023-04-17 12:59:06,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:06,966 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:06,967 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,967 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,974 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:06,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:06,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:06,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:06,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:06,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:06,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:06,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:06,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:06,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:06,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:06,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:06,994 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:06,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c, entries=150, sequenceid=290, filesize=11.9 K 2023-04-17 12:59:06,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:06,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:06,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/9da8caffc2c34eb3a4686e3afc23e532 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532 2023-04-17 12:59:07,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532, entries=150, sequenceid=290, filesize=11.9 K 2023-04-17 12:59:07,003 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 63889a78c730fa94931396cdfa19ec86 in 665ms, sequenceid=290, compaction requested=false 2023-04-17 12:59:07,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:07,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,005 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,012 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189f6ff9[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,012 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,013 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,013 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,013 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,013 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,013 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,014 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,014 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,015 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,016 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,016 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,016 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,017 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,017 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,017 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,019 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,020 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,021 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,021 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,020 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,021 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,066 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/a3e12072c0c04437a5d0572569310cf9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/a3e12072c0c04437a5d0572569310cf9 2023-04-17 12:59:07,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:07,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:07,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:07,078 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into a3e12072c0c04437a5d0572569310cf9(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:07,078 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:07,079 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=12, startTime=1681736346306; duration=0sec 2023-04-17 12:59:07,079 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:07,121 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,146 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,146 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,147 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,147 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,147 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,147 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:07,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,167 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,170 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,170 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,170 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,170 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,170 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,172 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,174 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e1f3c62[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,181 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,183 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,183 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,183 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,184 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,186 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,186 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,187 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,187 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,188 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,195 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,196 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,196 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736407213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736407213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,296 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,308 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,308 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,309 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,310 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,313 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,313 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,314 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,314 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,327 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,327 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,327 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,327 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,327 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,327 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,327 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,327 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,329 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736407323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736407326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a374f47[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,333 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,333 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,333 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,334 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,338 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,371 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,371 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,371 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,374 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,375 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,377 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,378 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,378 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,378 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,390 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,478 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,484 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,484 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,485 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,485 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:07,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,506 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,506 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,506 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,512 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,512 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,512 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,513 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,513 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,513 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,513 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,513 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,513 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,513 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,514 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,514 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,515 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,515 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547c7bb0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,520 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,520 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,523 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,523 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,523 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,523 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,523 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,523 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/534b5a46c73d45b8862f39664995b274 2023-04-17 12:59:07,527 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,527 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,527 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,527 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,531 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,531 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,532 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,535 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,541 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,541 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,541 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,542 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736407539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736407542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/8d300ddb249441cd9164d589d63fc8d3 2023-04-17 12:59:07,641 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,655 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,663 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,666 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:07,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,671 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,671 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,674 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,674 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,675 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,676 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,676 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377f3661[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,680 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,682 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,682 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,687 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,687 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,687 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,689 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,690 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,691 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,694 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,694 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:07,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,795 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,810 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:07,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,818 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,818 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,818 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,818 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,818 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,820 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50e4f0c0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,821 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,830 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,830 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,831 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,832 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,838 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,838 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,838 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736407849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:07,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736407849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,938 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:07,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:07,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:07,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:07,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:07,947 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:07,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:07,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:07,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,947 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,948 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,949 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,949 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:07,950 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:07,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:07,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:07,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:07,952 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,952 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:07,952 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:07,952 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:07,952 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:07,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:07,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:07,953 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:07,954 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:07,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6099d0ec[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:07,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:07,958 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:07,962 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,962 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:07,962 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:07,962 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:07,964 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:07,964 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,964 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:07,966 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:07,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:07,968 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:07,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:07,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:07,968 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:07,968 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:07,969 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,969 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,969 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:07,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:07,971 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,972 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,974 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:07,974 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:07,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:07,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:07,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:07,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:07,975 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:07,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:07,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:07,976 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:07,976 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:07,975 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,976 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:07,976 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,025 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/5c7589b2bdf8456ba1e9d22d434911df 2023-04-17 12:59:08,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/534b5a46c73d45b8862f39664995b274 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274 2023-04-17 12:59:08,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274, entries=200, sequenceid=308, filesize=14.3 K 2023-04-17 12:59:08,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/8d300ddb249441cd9164d589d63fc8d3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3 2023-04-17 12:59:08,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3, entries=150, sequenceid=308, filesize=11.9 K 2023-04-17 12:59:08,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/5c7589b2bdf8456ba1e9d22d434911df as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df 2023-04-17 12:59:08,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df, entries=150, sequenceid=308, filesize=11.9 K 2023-04-17 12:59:08,060 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 63889a78c730fa94931396cdfa19ec86 in 990ms, sequenceid=308, compaction requested=true 2023-04-17 12:59:08,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,061 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,061 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:08,062 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41848 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,062 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:08,062 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,062 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/b1eebb18ae5a4342a8c55843f33b8c9d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=40.9 K 2023-04-17 12:59:08,062 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,062 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:08,062 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,063 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b47c9b62bc6f4e8292a45861a0a66541, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.1 K 2023-04-17 12:59:08,063 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b1eebb18ae5a4342a8c55843f33b8c9d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:08,063 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b47c9b62bc6f4e8292a45861a0a66541, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:08,063 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 81bac9a92cd94ca48afb7ac981e804db, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681736345166 2023-04-17 12:59:08,063 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 534b5a46c73d45b8862f39664995b274, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,064 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 454bed70e4a8479fbb0a6d5dbe50d43c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681736345166 2023-04-17 12:59:08,064 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8d300ddb249441cd9164d589d63fc8d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,073 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,076 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:08,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:08,079 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:08,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:08,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:08,089 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:08,089 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,089 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:08,089 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:08,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,090 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,091 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,092 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:08,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:08,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:08,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:08,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,115 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:08,120 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/3ee4b3ae343347d98a50e3978746ac2e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ee4b3ae343347d98a50e3978746ac2e 2023-04-17 12:59:08,127 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:08,127 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:08,127 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,127 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:08,127 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 12:59:08,129 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:08,129 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:08,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:08,130 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,136 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 3ee4b3ae343347d98a50e3978746ac2e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:08,136 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,136 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=13, startTime=1681736348061; duration=0sec 2023-04-17 12:59:08,136 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:08,136 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,137 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,137 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:08,137 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,137 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/a3e12072c0c04437a5d0572569310cf9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.1 K 2023-04-17 12:59:08,138 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a3e12072c0c04437a5d0572569310cf9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681736343881 2023-04-17 12:59:08,139 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9da8caffc2c34eb3a4686e3afc23e532, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681736345166 2023-04-17 12:59:08,139 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5c7589b2bdf8456ba1e9d22d434911df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#399 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,157 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/a661f92acb5e4073924675a8a0494803 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a661f92acb5e4073924675a8a0494803 2023-04-17 12:59:08,167 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into a661f92acb5e4073924675a8a0494803(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:08,168 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,168 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=13, startTime=1681736348061; duration=0sec 2023-04-17 12:59:08,168 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,172 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f1f5d1765538475b9c11178b6fb97ec6 2023-04-17 12:59:08,202 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/e447a9eff47d4956b6e1c6650e5e7e15 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e447a9eff47d4956b6e1c6650e5e7e15 2023-04-17 12:59:08,208 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into e447a9eff47d4956b6e1c6650e5e7e15(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:08,208 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,208 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=13, startTime=1681736348061; duration=0sec 2023-04-17 12:59:08,208 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,222 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/bf5f7bdc22824581bf4ba00193443931 2023-04-17 12:59:08,252 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/4ec38121e3d743189b8bffb9049763b3 2023-04-17 12:59:08,265 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f1f5d1765538475b9c11178b6fb97ec6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6 2023-04-17 12:59:08,274 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6, entries=150, sequenceid=325, filesize=11.9 K 2023-04-17 12:59:08,275 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/bf5f7bdc22824581bf4ba00193443931 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931 2023-04-17 12:59:08,279 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931, entries=150, sequenceid=325, filesize=11.9 K 2023-04-17 12:59:08,281 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/4ec38121e3d743189b8bffb9049763b3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3 2023-04-17 12:59:08,292 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3, entries=150, sequenceid=325, filesize=11.9 K 2023-04-17 12:59:08,293 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 63889a78c730fa94931396cdfa19ec86 in 166ms, sequenceid=325, compaction requested=false 2023-04-17 12:59:08,293 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,293 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:08,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:08,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:08,293 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:08,296 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,297 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:08,300 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:08,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578f96b4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:08,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:08,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,304 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:08,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:08,304 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:08,304 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:08,305 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:08,305 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,305 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:08,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:08,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:08,309 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:08,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:08,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:08,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:08,310 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:08,316 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,316 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,316 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,318 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,324 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:08,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,329 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,329 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:08,329 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:08,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:08,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:08,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:08,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/275a2310ad984c49a36dbc70daffcee9 2023-04-17 12:59:08,434 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:08,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:08,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:08,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736348425, max timestamp=1681736348441Num uniques 859; 2023-04-17 12:59:08,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:08,452 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:08,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:08,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:08,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,455 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,456 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:08,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:08,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:08,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:08,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:08,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,466 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:08,468 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:08,468 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:08,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/907643caa3354075ad3ddb3b02af3a6b 2023-04-17 12:59:08,468 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,469 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:08,469 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:08,469 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:08,469 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:08,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,471 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:08,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736408471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:08,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190d6e1a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:08,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:08,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,481 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:08,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:08,481 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:08,481 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,481 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:08,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:08,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,482 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:08,486 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736408479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:08,496 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:08,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:08,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:08,496 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:08,496 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:08,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:08,497 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,497 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,498 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,499 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,500 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,500 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:08,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:08,500 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,501 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:08,501 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:08,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,502 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:08,515 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bd6765bdf94e4891b0117e4cdf146572 2023-04-17 12:59:08,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/275a2310ad984c49a36dbc70daffcee9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9 2023-04-17 12:59:08,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9, entries=200, sequenceid=338, filesize=14.3 K 2023-04-17 12:59:08,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/907643caa3354075ad3ddb3b02af3a6b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b 2023-04-17 12:59:08,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b, entries=150, sequenceid=338, filesize=11.9 K 2023-04-17 12:59:08,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bd6765bdf94e4891b0117e4cdf146572 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572 2023-04-17 12:59:08,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572, entries=150, sequenceid=338, filesize=11.9 K 2023-04-17 12:59:08,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=129.71 KB/132820 for 63889a78c730fa94931396cdfa19ec86 in 169ms, sequenceid=338, compaction requested=true 2023-04-17 12:59:08,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,549 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,549 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:08,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:08,551 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,551 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,551 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:08,551 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:08,552 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,552 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,552 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ee4b3ae343347d98a50e3978746ac2e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.2 K 2023-04-17 12:59:08,552 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a661f92acb5e4073924675a8a0494803, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=38.6 K 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3ee4b3ae343347d98a50e3978746ac2e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting a661f92acb5e4073924675a8a0494803, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting bf5f7bdc22824581bf4ba00193443931, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681736347190 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f1f5d1765538475b9c11178b6fb97ec6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681736347190 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 907643caa3354075ad3ddb3b02af3a6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348371 2023-04-17 12:59:08,552 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 275a2310ad984c49a36dbc70daffcee9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348364 2023-04-17 12:59:08,558 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,570 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,601 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:08,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:08,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:08,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:08,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:08,606 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:08,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:08,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:08,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,606 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=143.13 KB heapSize=421.48 KB 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:08,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:08,607 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:08,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,607 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:08,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:08,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:08,609 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:08,609 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,609 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:08,609 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:08,609 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:08,610 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,610 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:08,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:08,613 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:08,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:08,613 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,615 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:08,615 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:08,623 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:08,633 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/64ec4402b350440c9e44c2838fbc532c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/64ec4402b350440c9e44c2838fbc532c 2023-04-17 12:59:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:08,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e98ff0b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:08,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:08,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,653 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,653 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:08,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:08,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:08,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:08,659 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:08,659 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,659 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,659 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:08,659 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into 64ec4402b350440c9e44c2838fbc532c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,659 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:08,659 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=13, startTime=1681736348549; duration=0sec 2023-04-17 12:59:08,659 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:08,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,659 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,660 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:08,661 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:08,661 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,661 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e447a9eff47d4956b6e1c6650e5e7e15, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.2 K 2023-04-17 12:59:08,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,661 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e447a9eff47d4956b6e1c6650e5e7e15, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736346400 2023-04-17 12:59:08,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,662 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 4ec38121e3d743189b8bffb9049763b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681736347190 2023-04-17 12:59:08,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:08,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:08,662 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:08,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:08,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:08,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:08,662 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:08,663 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting bd6765bdf94e4891b0117e4cdf146572, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348371 2023-04-17 12:59:08,664 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,665 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ca8904e26d0848ddae8780195123b4a5 2023-04-17 12:59:08,667 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,667 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,672 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:08,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:08,672 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:08,673 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:08,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:08,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,675 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:08,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:08,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736408679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736408691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,774 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:08,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:08,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:08,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:08,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:08,801 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:08,801 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:08,802 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:08,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736408791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,803 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,804 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,804 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,807 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,807 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:08,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:08,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:08,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:08,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:08,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:08,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,808 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:08,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:08,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:08,814 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:08,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:08,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:08,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736408806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,820 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,820 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,822 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:08,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:08,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a84c84[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:08,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:08,825 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,826 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:08,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:08,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:08,826 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,826 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:08,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:08,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,827 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:08,827 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:08,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:08,830 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:08,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:08,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:08,830 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:08,830 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:08,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,833 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:08,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:08,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,837 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:08,847 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,847 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:08,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:08,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:08,850 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:08,850 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:08,954 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:08,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:08,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:08,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:08,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:08,988 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:08,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:08,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:08,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,989 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:08,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:08,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:08,993 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:08,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:08,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:08,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:08,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:08,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:08,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,001 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,002 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,002 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,003 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/6110ba6f5427416789eb7ffc47abda9a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6110ba6f5427416789eb7ffc47abda9a 2023-04-17 12:59:09,003 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,003 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,004 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb95813[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,007 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,007 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,008 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,009 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,009 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,009 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,010 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:09,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,013 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,013 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,013 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736409007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736409018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,023 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,023 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,024 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,024 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 6110ba6f5427416789eb7ffc47abda9a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:09,024 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,024 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=13, startTime=1681736348549; duration=0sec 2023-04-17 12:59:09,024 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,027 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,040 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,042 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,043 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,043 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,043 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:09,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/95ad34f703f3461fa7ccfd19312709d8 2023-04-17 12:59:09,143 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:09,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:09,146 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/419b264065dc42cf8bcff32559cdc64b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/419b264065dc42cf8bcff32559cdc64b 2023-04-17 12:59:09,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:09,157 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into 419b264065dc42cf8bcff32559cdc64b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:09,157 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,158 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=13, startTime=1681736348550; duration=0sec 2023-04-17 12:59:09,158 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:09,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:09,162 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:09,162 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,162 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:09,162 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:09,166 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,166 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,167 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:09,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:09,168 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,169 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,169 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:09,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:09,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:09,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:09,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,181 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,196 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,196 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,197 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,197 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,197 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,197 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,198 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/5b8665f320e44b608a0707e3f3ccd9b9 2023-04-17 12:59:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ca8904e26d0848ddae8780195123b4a5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5 2023-04-17 12:59:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b686c0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,227 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,228 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,228 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,228 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,229 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5, entries=250, sequenceid=367, filesize=16.7 K 2023-04-17 12:59:09,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/95ad34f703f3461fa7ccfd19312709d8 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8 2023-04-17 12:59:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:09,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,244 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,244 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,245 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,245 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,245 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,247 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,249 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,252 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,252 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,253 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,253 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,255 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,255 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:09,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 12:59:09,268 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-17 12:59:09,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/5b8665f320e44b608a0707e3f3ccd9b9 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9 2023-04-17 12:59:09,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 12:59:09,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~421.44 KB/431552, currentSize=33.54 KB/34350 for 63889a78c730fa94931396cdfa19ec86 in 680ms, sequenceid=367, compaction requested=false 2023-04-17 12:59:09,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:09,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:59:09,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:09,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:09,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:09,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:09,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:09,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:09,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/babca5d1524343f98f268ccc9913120a 2023-04-17 12:59:09,355 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:09,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:09,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:09,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:09,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:09,360 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:09,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:09,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:09,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,361 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,362 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:09,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:09,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:09,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:09,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:09,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,381 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,381 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,381 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,381 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,384 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ff525a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,386 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,387 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,387 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,388 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,388 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,389 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,389 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:09,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,391 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,391 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,393 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9c6022535dc745f9b8a78709e62cbfe1 2023-04-17 12:59:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,394 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,401 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,402 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,402 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:09,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/60a35daae5364bbe80967799da5948fb 2023-04-17 12:59:09,430 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/babca5d1524343f98f268ccc9913120a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a 2023-04-17 12:59:09,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a, entries=200, sequenceid=380, filesize=14.3 K 2023-04-17 12:59:09,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/9c6022535dc745f9b8a78709e62cbfe1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1 2023-04-17 12:59:09,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1, entries=150, sequenceid=380, filesize=11.9 K 2023-04-17 12:59:09,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/60a35daae5364bbe80967799da5948fb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb 2023-04-17 12:59:09,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736349413, max timestamp=1681736349451Num uniques 859; 2023-04-17 12:59:09,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:59:09,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736349413, max timestamp=1681736349451Num uniques 859; 2023-04-17 12:59:09,463 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:59:09,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736409455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736349413, max timestamp=1681736349451Num uniques 859; 2023-04-17 12:59:09,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:59:09,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb, entries=150, sequenceid=380, filesize=11.9 K 2023-04-17 12:59:09,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=39.07 KB/40005 for 63889a78c730fa94931396cdfa19ec86 in 147ms, sequenceid=380, compaction requested=true 2023-04-17 12:59:09,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:09,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:09,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:09,481 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:09,481 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:09,482 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44492 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:09,482 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:09,483 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,483 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/64ec4402b350440c9e44c2838fbc532c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=43.4 K 2023-04-17 12:59:09,483 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:09,483 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:09,483 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,483 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6110ba6f5427416789eb7ffc47abda9a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.3 K 2023-04-17 12:59:09,484 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 64ec4402b350440c9e44c2838fbc532c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348371 2023-04-17 12:59:09,484 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ca8904e26d0848ddae8780195123b4a5, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681736348467 2023-04-17 12:59:09,484 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 6110ba6f5427416789eb7ffc47abda9a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348371 2023-04-17 12:59:09,484 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting babca5d1524343f98f268ccc9913120a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348674 2023-04-17 12:59:09,485 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 95ad34f703f3461fa7ccfd19312709d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681736348476 2023-04-17 12:59:09,485 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9c6022535dc745f9b8a78709e62cbfe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348680 2023-04-17 12:59:09,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=52.49 KB heapSize=143.18 KB 2023-04-17 12:59:09,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:09,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:09,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:09,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:09,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:09,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:09,495 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:09,502 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:09,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:09,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:09,520 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:09,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:09,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:09,521 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:09,521 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,521 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:09,521 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:09,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,532 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,542 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,543 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:09,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:09,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:09,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:09,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:09,557 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:09,557 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,557 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,557 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,557 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,558 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,558 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,566 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,566 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,569 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,570 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.50 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ade875c52a5748ffba3107517f6047a0 2023-04-17 12:59:09,571 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,574 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,574 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b1e4ad[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,589 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,589 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,589 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,589 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,591 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,591 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,594 WARN [DataStreamer for file /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/2e6368e9f5004138ba14ab5918e833eb] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-17 12:59:09,600 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/1779b00a95ea4cd7ae9e2ec0f930fb43 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/1779b00a95ea4cd7ae9e2ec0f930fb43 2023-04-17 12:59:09,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,600 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,606 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,624 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into 1779b00a95ea4cd7ae9e2ec0f930fb43(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:09,625 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,625 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=13, startTime=1681736349479; duration=0sec 2023-04-17 12:59:09,625 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:09,625 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,630 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:09,630 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:09,630 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,630 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/419b264065dc42cf8bcff32559cdc64b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.3 K 2023-04-17 12:59:09,630 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/2e6368e9f5004138ba14ab5918e833eb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/2e6368e9f5004138ba14ab5918e833eb 2023-04-17 12:59:09,634 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 419b264065dc42cf8bcff32559cdc64b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681736348371 2023-04-17 12:59:09,635 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,636 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 5b8665f320e44b608a0707e3f3ccd9b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681736348476 2023-04-17 12:59:09,636 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 60a35daae5364bbe80967799da5948fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348680 2023-04-17 12:59:09,642 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 2e6368e9f5004138ba14ab5918e833eb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:09,642 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:09,642 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=13, startTime=1681736349481; duration=0sec 2023-04-17 12:59:09,642 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:09,646 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,648 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,649 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,650 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,650 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:09,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736349645, max timestamp=1681736349679Num uniques 859; 2023-04-17 12:59:09,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:59:09,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736349645, max timestamp=1681736349679Num uniques 859; 2023-04-17 12:59:09,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:59:09,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736349645, max timestamp=1681736349679Num uniques 859; 2023-04-17 12:59:09,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:09,754 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:09,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:09,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:09,758 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:09,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:09,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:09,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,770 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,774 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,775 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,775 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:09,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:09,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:09,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:09,784 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:09,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736409768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736409780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,789 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e95b28[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,792 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,793 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,793 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,795 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,795 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,797 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,805 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,805 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,805 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,805 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,806 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,806 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:09,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736409888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:09,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736409894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,906 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:09,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:09,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:09,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:09,917 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:09,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:09,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:09,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:09,921 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:09,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:09,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:09,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:09,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:09,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:09,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:09,925 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:09,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:09,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:09,926 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:09,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:09,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:09,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:09,926 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:09,927 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,928 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e445f13[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:09,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:09,933 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,934 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:09,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:09,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,934 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:09,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:09,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:09,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,935 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,935 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:09,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:09,937 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:09,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:09,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:09,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:09,938 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:09,939 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,939 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:09,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,939 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,940 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:09,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:09,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:09,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:09,943 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:09,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:09,944 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:09,944 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:10,045 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:10,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:10,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:10,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.50 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/45b05eb7aef942b0b0fc96019aa75dbc 2023-04-17 12:59:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:10,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:10,051 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:10,052 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,052 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:10,052 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:10,052 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,052 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,053 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,053 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,054 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:10,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:10,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:10,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:10,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:10,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,057 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:10,062 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:10,062 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:10,063 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,063 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:10,063 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:10,063 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,063 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:10,063 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:10,063 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,063 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:10,063 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:10,065 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,065 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,065 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:10,066 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.50 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/b3709806892646d9a0748cc4e3c9f91c 2023-04-17 12:59:10,070 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/13acebe3b372429eb1fc6259eaa785cb as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/13acebe3b372429eb1fc6259eaa785cb 2023-04-17 12:59:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ade875c52a5748ffba3107517f6047a0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0 2023-04-17 12:59:10,076 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into 13acebe3b372429eb1fc6259eaa785cb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:10,076 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:10,076 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=13, startTime=1681736349481; duration=0sec 2023-04-17 12:59:10,076 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0, entries=200, sequenceid=405, filesize=14.3 K 2023-04-17 12:59:10,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:10,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7463b703[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:10,085 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:10,085 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/45b05eb7aef942b0b0fc96019aa75dbc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc 2023-04-17 12:59:10,086 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,086 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:10,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:10,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:10,086 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:10,087 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:10,087 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,087 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:10,090 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:10,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:10,093 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:10,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:10,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:10,093 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:10,093 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:10,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 12:59:10,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,095 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,097 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/b3709806892646d9a0748cc4e3c9f91c as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c 2023-04-17 12:59:10,098 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,098 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,101 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,101 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,102 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:10,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:10,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:10,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 12:59:10,105 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:10,105 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:10,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~52.49 KB/53745, heapSize ~143.13 KB/146568, currentSize=131.94 KB/135110 for 63889a78c730fa94931396cdfa19ec86 in 616ms, sequenceid=405, compaction requested=false 2023-04-17 12:59:10,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:10,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:10,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:10,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:10,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/2c8c2e945b3a4898b85868dd3a2f5584 2023-04-17 12:59:10,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736410135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736410153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,205 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:10,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:10,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:10,216 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:10,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:10,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:10,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,217 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,217 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:10,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:10,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:10,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:10,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:10,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,226 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:10,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:10,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:10,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:10,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:10,230 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:10,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:10,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:10,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:10,232 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,232 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,233 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:10,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@494aa1ae[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:10,237 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:10,237 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,240 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,240 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:10,240 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:10,240 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:10,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,243 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:10,243 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:10,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,243 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:10,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:10,247 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:10,247 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:10,247 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:10,247 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:10,247 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:10,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,249 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,252 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,256 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,256 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:10,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:10,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:10,258 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:10,258 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:10,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736410254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736410275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,358 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:10,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:10,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:10,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:10,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:10,363 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:10,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:10,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:10,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,364 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:10,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:10,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:10,367 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:10,367 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,367 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:10,367 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:10,367 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:10,367 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,367 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:10,368 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:10,368 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,368 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:10,368 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:10,368 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,368 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:10,368 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:10,370 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,371 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@713a08da[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:10,385 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,390 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,391 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:10,392 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:10,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:10,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:10,395 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:10,395 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,395 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,395 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:10,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:10,411 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:10,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:10,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:10,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:10,411 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:10,412 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,412 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,413 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,414 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,417 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,421 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,421 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,422 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,422 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:10,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:10,423 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:10,423 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:10,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736410464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736410480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,525 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:10,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:10,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:10,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:10,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:10,534 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:10,534 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,534 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:10,534 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:10,535 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,535 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,538 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:10,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:10,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:10,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:10,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:10,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:10,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:10,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:10,549 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:10,551 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,551 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:10,551 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:10,551 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,551 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:10,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:10,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:10,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:10,553 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,555 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,555 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:10,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102393b4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:10,559 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:10,559 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:10,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:10,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:10,561 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:10,561 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,561 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:10,561 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:10,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:10,562 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:10,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:10,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:10,562 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:10,562 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:10,563 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,563 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,563 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:10,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:10,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,564 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,571 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:10,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,582 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:10,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:10,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:10,584 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:10,585 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:10,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:10,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/7d48c5261c084848817eaacfa6dd8955 2023-04-17 12:59:10,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=445 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/7aa85e4da5ae4966ad9cdb44b4baabe7 2023-04-17 12:59:10,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/2c8c2e945b3a4898b85868dd3a2f5584 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584 2023-04-17 12:59:10,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584, entries=200, sequenceid=445, filesize=14.3 K 2023-04-17 12:59:10,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/7d48c5261c084848817eaacfa6dd8955 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955 2023-04-17 12:59:10,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955, entries=150, sequenceid=445, filesize=11.9 K 2023-04-17 12:59:10,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/7aa85e4da5ae4966ad9cdb44b4baabe7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7 2023-04-17 12:59:10,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7, entries=150, sequenceid=445, filesize=11.9 K 2023-04-17 12:59:10,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 63889a78c730fa94931396cdfa19ec86 in 547ms, sequenceid=445, compaction requested=true 2023-04-17 12:59:10,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:10,660 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:10,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:59:10,660 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:10,662 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:10,662 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:10,662 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:10,662 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,662 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/1779b00a95ea4cd7ae9e2ec0f930fb43, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=41.2 K 2023-04-17 12:59:10,662 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:10,662 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,662 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/2e6368e9f5004138ba14ab5918e833eb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.4 K 2023-04-17 12:59:10,662 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1779b00a95ea4cd7ae9e2ec0f930fb43, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348680 2023-04-17 12:59:10,663 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 2e6368e9f5004138ba14ab5918e833eb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348680 2023-04-17 12:59:10,663 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ade875c52a5748ffba3107517f6047a0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681736349413 2023-04-17 12:59:10,663 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 45b05eb7aef942b0b0fc96019aa75dbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681736349413 2023-04-17 12:59:10,663 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2c8c2e945b3a4898b85868dd3a2f5584, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:10,664 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7d48c5261c084848817eaacfa6dd8955, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:10,673 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:10,674 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:10,689 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:10,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:10,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:10,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:10,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:10,703 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:10,703 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,703 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:10,703 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:10,715 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,715 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,716 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,718 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:10,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:10,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:10,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:10,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:10,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:10,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:10,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:10,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:10,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:10,738 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:10,738 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:10,738 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:10,738 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:10,739 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:10,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:10,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:10,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736350849, max timestamp=1681736350883Num uniques 859; 2023-04-17 12:59:10,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:59:10,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736350849, max timestamp=1681736350883Num uniques 859; 2023-04-17 12:59:10,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:59:10,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681736350849, max timestamp=1681736350883Num uniques 859; 2023-04-17 12:59:10,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 12:59:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736410957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736410958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736411067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736411074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,107 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ba223147ec90453485a6a10b38262ab7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ba223147ec90453485a6a10b38262ab7 2023-04-17 12:59:11,107 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/99cfa0f59fd94e43b1a8cdd3d17353d1 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/99cfa0f59fd94e43b1a8cdd3d17353d1 2023-04-17 12:59:11,114 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into ba223147ec90453485a6a10b38262ab7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:11,114 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into 99cfa0f59fd94e43b1a8cdd3d17353d1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:11,114 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:11,114 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:11,114 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=13, startTime=1681736350660; duration=0sec 2023-04-17 12:59:11,114 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:11,114 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:11,114 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=13, startTime=1681736350660; duration=0sec 2023-04-17 12:59:11,114 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:11,115 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:11,115 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:11,115 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:11,115 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/13acebe3b372429eb1fc6259eaa785cb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=36.4 K 2023-04-17 12:59:11,116 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 13acebe3b372429eb1fc6259eaa785cb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681736348680 2023-04-17 12:59:11,116 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b3709806892646d9a0748cc4e3c9f91c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681736349413 2023-04-17 12:59:11,116 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 7aa85e4da5ae4966ad9cdb44b4baabe7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:11,154 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:11,161 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/ffb7f5a254b24a7987f5b7671bda01ba as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/ffb7f5a254b24a7987f5b7671bda01ba 2023-04-17 12:59:11,166 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into ffb7f5a254b24a7987f5b7671bda01ba(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:11,166 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:11,166 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=13, startTime=1681736350660; duration=0sec 2023-04-17 12:59:11,166 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:11,198 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/35e36893df69480ca9f072e687c4b57d 2023-04-17 12:59:11,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736411278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736411294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736411589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:11,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736411607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:11,663 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/ae0adfa9abec4957bd38e04df9b67578 2023-04-17 12:59:12,100 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/e41c9307858844a290330b2fb7ac025f 2023-04-17 12:59:12,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:12,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736412097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,105 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/35e36893df69480ca9f072e687c4b57d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d 2023-04-17 12:59:12,108 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d, entries=100, sequenceid=454, filesize=9.5 K 2023-04-17 12:59:12,109 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/ae0adfa9abec4957bd38e04df9b67578 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578 2023-04-17 12:59:12,113 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578, entries=100, sequenceid=454, filesize=9.5 K 2023-04-17 12:59:12,118 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/e41c9307858844a290330b2fb7ac025f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f 2023-04-17 12:59:12,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:12,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736412113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,125 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f, entries=100, sequenceid=454, filesize=9.5 K 2023-04-17 12:59:12,126 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 63889a78c730fa94931396cdfa19ec86 in 1388ms, sequenceid=454, compaction requested=false 2023-04-17 12:59:12,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,126 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,126 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,127 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,128 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,128 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,128 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c83fa64[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,131 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,132 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,132 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,132 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,132 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,135 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,135 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,135 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,136 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,136 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,136 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,138 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,139 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,141 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,141 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,141 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,141 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,241 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:12,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:12,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:12,244 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:12,244 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:12,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:12,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,245 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,246 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,246 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:12,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:12,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:12,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:12,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:12,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:12,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:12,247 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:12,247 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,247 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:12,247 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:12,248 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:12,277 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/9d7494fb30d145409ba387a145a2b764 2023-04-17 12:59:12,303 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/b85bf2bc608a427db4551f86ae20bd4e 2023-04-17 12:59:12,360 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/02c72b4afca2430092d319f944118a06 2023-04-17 12:59:12,364 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/9d7494fb30d145409ba387a145a2b764 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764 2023-04-17 12:59:12,367 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 12:59:12,368 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/b85bf2bc608a427db4551f86ae20bd4e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e 2023-04-17 12:59:12,375 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 12:59:12,376 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/02c72b4afca2430092d319f944118a06 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06 2023-04-17 12:59:12,381 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 12:59:12,381 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for 63889a78c730fa94931396cdfa19ec86 in 134ms, sequenceid=494, compaction requested=true 2023-04-17 12:59:12,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,381 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,381 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,382 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65319f58[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,401 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,401 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,401 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,402 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,404 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,405 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,406 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,406 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,406 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,406 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,408 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,410 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,411 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,411 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:12,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,511 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:12,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:12,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:12,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:12,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:12,528 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:12,528 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,528 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:12,528 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:12,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,530 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:12,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:12,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:12,532 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:12,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,533 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:12,533 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,533 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:12,533 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,533 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,533 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,535 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,535 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,535 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,535 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ea1581[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,539 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,539 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,539 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,540 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,540 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,540 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,541 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,546 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,546 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,546 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,549 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,550 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,551 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,553 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,555 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,555 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,555 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,555 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,555 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:12,655 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:12,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:12,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:12,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:12,660 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:12,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:12,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:12,660 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,661 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,661 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,661 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:12,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:12,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:12,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:12,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:12,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:12,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:12,663 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:12,663 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,664 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:12,664 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,664 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,664 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,665 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,665 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d8973b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,668 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,669 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,669 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,675 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,675 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,677 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,677 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,678 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,681 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,682 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,684 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,684 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,684 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,684 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,685 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:12,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,784 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:12,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:12,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:12,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:12,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:12,790 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:12,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:12,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:12,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,794 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:12,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:12,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:12,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:12,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:12,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:12,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:12,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,797 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:12,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,797 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,797 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,798 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@661b033e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,800 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,800 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,801 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,801 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,801 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,801 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,801 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,802 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,804 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,805 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,805 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,806 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,807 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,809 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,812 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,812 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,812 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:12,913 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:12,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:12,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:12,916 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:12,916 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:12,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:12,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,917 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,918 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:12,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:12,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:12,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:12,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:12,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:12,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:12,919 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,919 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:12,919 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:12,920 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:12,920 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:12,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:12,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:12,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:12,920 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:12,921 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,921 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fb99bfa[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:12,925 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:12,925 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,926 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:12,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:12,927 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:12,928 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,928 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:12,928 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:12,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,929 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:12,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:12,935 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:12,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:12,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:12,935 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:12,935 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:12,936 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,936 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,936 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,937 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,939 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:12,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:12,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:12,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:12,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:12,942 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:12,942 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:12,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:12,941 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:12,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:12,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,042 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,058 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,058 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,058 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,058 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,059 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,059 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,060 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,060 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,060 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,061 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,066 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,066 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,066 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:13,066 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,067 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,067 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,068 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26d6c191[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,072 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,072 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,072 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,073 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,073 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:13,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,076 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,076 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,076 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,076 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,076 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,077 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,079 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,080 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,083 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,084 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,084 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,083 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,083 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,083 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:13,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:13,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:13,184 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,190 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,192 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,193 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,193 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,193 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1786eb3f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,199 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,199 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,199 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,200 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,200 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,200 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,200 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:13,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,203 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,203 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,203 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,207 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,207 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,207 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,212 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736353178, max timestamp=1681736353213Num uniques 859; 2023-04-17 12:59:13,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:13,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736353178, max timestamp=1681736353213Num uniques 859; 2023-04-17 12:59:13,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:13,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736353178, max timestamp=1681736353213Num uniques 859; 2023-04-17 12:59:13,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,221 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,221 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,221 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,222 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736413282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736413286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,321 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,326 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,326 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,327 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,328 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,328 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,329 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,330 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,331 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,331 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,331 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,331 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,332 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,332 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293a1802[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,334 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,335 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,335 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,335 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,336 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,336 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,336 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,336 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:13,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,340 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,340 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,340 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,341 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,341 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,342 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,342 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,342 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,345 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,346 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,347 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,348 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,348 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736413393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736413394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,448 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,453 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,454 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,454 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,455 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,462 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,462 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,462 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,462 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,462 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,467 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,467 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9f2094[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,471 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,472 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,472 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,474 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,474 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,474 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,476 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,477 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,479 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,483 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,483 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,483 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,483 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,484 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/076d66faafcc41708dc9ae292b48dc47 2023-04-17 12:59:13,584 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,590 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,590 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,591 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,592 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,592 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,594 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,594 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,597 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/6cd1513f95774ce18fa079d401da6729 2023-04-17 12:59:13,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad5be1f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,600 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,601 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,601 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,601 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,601 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,602 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,603 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,603 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,603 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,603 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,605 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,606 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,607 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,610 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,616 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,617 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,617 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,616 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,617 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,617 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,617 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,617 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,616 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,618 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736413607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736413607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,718 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,727 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,727 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,735 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,735 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,741 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,742 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,743 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,743 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,744 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,744 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,744 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,744 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,744 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,748 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,749 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,749 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,749 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2a6d69[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,755 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,758 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,758 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,760 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,761 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,761 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,761 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:13,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,761 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:13,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:13,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:13,761 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:13,761 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:13,766 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,766 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,766 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,773 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,784 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,803 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:13,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:13,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,803 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:13,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:13,804 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:13,804 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:13,904 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:13,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:13,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:13,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:13,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:13,908 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:13,908 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,908 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:13,908 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:13,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,915 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,927 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:13,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:13,928 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:13,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:13,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:13,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39068 deadline: 1681736413919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:13,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:39052 deadline: 1681736413928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:13,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:13,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:13,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:13,934 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:13,954 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:13,954 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:13,954 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,954 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:13,954 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:13,954 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:13,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:13,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:13,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:13,956 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:13,957 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:13,962 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:13,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:13,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:13,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:13,964 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23a98426[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:13,964 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,965 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:13,979 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:13,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:13,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:13,990 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:13,992 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:13,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:14,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:14,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:14,054 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:14,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:14,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:14,055 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:14,055 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:14,065 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,065 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:14,065 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:14,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/603629ad92d446f890ae3e1671a10f6e 2023-04-17 12:59:14,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:14,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:14,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:14,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/076d66faafcc41708dc9ae292b48dc47 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47 2023-04-17 12:59:14,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:14,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:14,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:14,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:14,130 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:14,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:14,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:14,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:14,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:14,134 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:14,134 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:14,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:14,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47, entries=200, sequenceid=509, filesize=14.3 K 2023-04-17 12:59:14,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/6cd1513f95774ce18fa079d401da6729 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729 2023-04-17 12:59:14,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729, entries=150, sequenceid=509, filesize=11.9 K 2023-04-17 12:59:14,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/603629ad92d446f890ae3e1671a10f6e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e 2023-04-17 12:59:14,236 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:14,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:14,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e, entries=150, sequenceid=509, filesize=11.9 K 2023-04-17 12:59:14,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:14,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 63889a78c730fa94931396cdfa19ec86 in 1150ms, sequenceid=509, compaction requested=true 2023-04-17 12:59:14,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:14,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:14,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:59:14,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:59:14,285 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:14,286 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:14,291 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:14,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:14,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:14,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:14,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,303 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:14,306 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:14,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:14,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:14,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:14,310 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:14,314 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:14,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:14,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:14,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:14,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:14,322 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:14,322 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:14,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:14,326 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/C is initiating minor compaction (all files) 2023-04-17 12:59:14,326 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/C in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:14,326 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/ffb7f5a254b24a7987f5b7671bda01ba, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=46.0 K 2023-04-17 12:59:14,326 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:14,326 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:14,330 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. started... 2023-04-17 12:59:14,330 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49558 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:14,330 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/A is initiating minor compaction (all files) 2023-04-17 12:59:14,330 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/A in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:14,330 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ba223147ec90453485a6a10b38262ab7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=48.4 K 2023-04-17 12:59:14,334 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ba223147ec90453485a6a10b38262ab7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:14,335 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ffb7f5a254b24a7987f5b7671bda01ba, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:14,330 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:59:14,335 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:14,335 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:14,335 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:14,336 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:14,336 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:14,336 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:14,341 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 35e36893df69480ca9f072e687c4b57d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681736350132 2023-04-17 12:59:14,342 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e41c9307858844a290330b2fb7ac025f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681736350132 2023-04-17 12:59:14,346 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 9d7494fb30d145409ba387a145a2b764, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681736350939 2023-04-17 12:59:14,346 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 02c72b4afca2430092d319f944118a06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681736350939 2023-04-17 12:59:14,355 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 076d66faafcc41708dc9ae292b48dc47, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681736353121 2023-04-17 12:59:14,358 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 603629ad92d446f890ae3e1671a10f6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681736353123 2023-04-17 12:59:14,426 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f2fb9d7a44f441f4bcde0cdf1fe71369 2023-04-17 12:59:14,430 DEBUG [Thread-2608] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x209faa3c to 127.0.0.1:62647 2023-04-17 12:59:14,430 DEBUG [Thread-2608] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,431 DEBUG [Thread-2604] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ea9d51a to 127.0.0.1:62647 2023-04-17 12:59:14,431 DEBUG [Thread-2604] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,432 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#A#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:14,433 DEBUG [Thread-2610] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5345748a to 127.0.0.1:62647 2023-04-17 12:59:14,433 DEBUG [Thread-2610] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:14,434 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. as already flushing 2023-04-17 12:59:14,434 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#C#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:14,434 DEBUG [Thread-2599] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58ae18e2 to 127.0.0.1:62647 2023-04-17 12:59:14,435 DEBUG [Thread-2599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,439 DEBUG [Thread-2593] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ff7cd4 to 127.0.0.1:62647 2023-04-17 12:59:14,439 DEBUG [Thread-2593] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,444 DEBUG [Thread-2606] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fd1c1f to 127.0.0.1:62647 2023-04-17 12:59:14,444 DEBUG [Thread-2606] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,453 DEBUG [Thread-2602] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:62647 2023-04-17 12:59:14,453 DEBUG [Thread-2602] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:14,853 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f738b8d62db4473e8322073ea58c7f89 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f738b8d62db4473e8322073ea58c7f89 2023-04-17 12:59:14,857 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/779ee4b284b44253b924531832ab6c61 2023-04-17 12:59:14,862 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/A of 63889a78c730fa94931396cdfa19ec86 into f738b8d62db4473e8322073ea58c7f89(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:14,862 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:14,862 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/A, priority=12, startTime=1681736354284; duration=0sec 2023-04-17 12:59:14,862 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:14,862 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:14,864 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47118 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:14,864 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 63889a78c730fa94931396cdfa19ec86/B is initiating minor compaction (all files) 2023-04-17 12:59:14,864 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63889a78c730fa94931396cdfa19ec86/B in TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:14,864 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/99cfa0f59fd94e43b1a8cdd3d17353d1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp, totalSize=46.0 K 2023-04-17 12:59:14,865 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 99cfa0f59fd94e43b1a8cdd3d17353d1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=445, earliestPutTs=1681736349744 2023-04-17 12:59:14,865 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ae0adfa9abec4957bd38e04df9b67578, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681736350132 2023-04-17 12:59:14,872 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b85bf2bc608a427db4551f86ae20bd4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681736350939 2023-04-17 12:59:14,887 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6cd1513f95774ce18fa079d401da6729, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681736353123 2023-04-17 12:59:14,918 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/3e9f830b958a41fb90c32b1e0d7a3c70 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3e9f830b958a41fb90c32b1e0d7a3c70 2023-04-17 12:59:14,923 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/C of 63889a78c730fa94931396cdfa19ec86 into 3e9f830b958a41fb90c32b1e0d7a3c70(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:14,923 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:14,923 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/C, priority=12, startTime=1681736354285; duration=0sec 2023-04-17 12:59:14,923 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:14,924 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63889a78c730fa94931396cdfa19ec86#B#compaction#440 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:14,990 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/4e067cfa1a914389971566c27115cefe 2023-04-17 12:59:15,082 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/bad04f08edc7477aaba0723d0cc585cd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bad04f08edc7477aaba0723d0cc585cd 2023-04-17 12:59:15,094 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/f2fb9d7a44f441f4bcde0cdf1fe71369 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f2fb9d7a44f441f4bcde0cdf1fe71369 2023-04-17 12:59:15,150 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63889a78c730fa94931396cdfa19ec86/B of 63889a78c730fa94931396cdfa19ec86 into bad04f08edc7477aaba0723d0cc585cd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:15,151 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:15,151 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86., storeName=63889a78c730fa94931396cdfa19ec86/B, priority=12, startTime=1681736354285; duration=0sec 2023-04-17 12:59:15,151 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:15,154 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f2fb9d7a44f441f4bcde0cdf1fe71369, entries=150, sequenceid=546, filesize=11.9 K 2023-04-17 12:59:15,156 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/779ee4b284b44253b924531832ab6c61 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/779ee4b284b44253b924531832ab6c61 2023-04-17 12:59:15,162 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/779ee4b284b44253b924531832ab6c61, entries=150, sequenceid=546, filesize=11.9 K 2023-04-17 12:59:15,163 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/4e067cfa1a914389971566c27115cefe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4e067cfa1a914389971566c27115cefe 2023-04-17 12:59:15,168 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4e067cfa1a914389971566c27115cefe, entries=150, sequenceid=546, filesize=11.9 K 2023-04-17 12:59:15,169 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=13.42 KB/13740 for 63889a78c730fa94931396cdfa19ec86 in 839ms, sequenceid=546, compaction requested=false 2023-04-17 12:59:15,169 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:15,169 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:15,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:15,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:15,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:15,169 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:15,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,171 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:15,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,171 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:15,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:15,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a18b47[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:15,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:15,176 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,177 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,177 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:15,177 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,177 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:15,177 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:15,177 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:15,178 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:15,178 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:15,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:15,179 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:15,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:15,181 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:15,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:15,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:15,181 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:15,181 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:15,183 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,183 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:15,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,184 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:15,185 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:15,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,187 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,192 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:15,192 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:15,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:15,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,195 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:15,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:15,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:15,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:15,196 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:15,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:15,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:15,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:15,197 DEBUG [Thread-2601] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:15,197 DEBUG [Thread-2601] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:15,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:15,297 DEBUG [Thread-2601] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:15,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:15,813 DEBUG [Thread-2595] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06846250 to 127.0.0.1:62647 2023-04-17 12:59:15,813 DEBUG [Thread-2595] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:15,893 DEBUG [Thread-2591] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ea7ac79 to 127.0.0.1:62647 2023-04-17 12:59:15,893 DEBUG [Thread-2591] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:15,948 DEBUG [Thread-2597] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x647f95f6 to 127.0.0.1:62647 2023-04-17 12:59:15,948 DEBUG [Thread-2597] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 176 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 180 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1122 2023-04-17 12:59:15,948 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3360 rows 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1122 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3366 rows 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1116 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3348 rows 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1116 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3342 rows 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(402): scanned 1120 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(403): verified 3360 rows 2023-04-17 12:59:15,949 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:59:15,949 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e895a5d to 127.0.0.1:62647 2023-04-17 12:59:15,949 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:15,958 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:59:15,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:59:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:15,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:15,972 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736355972"}]},"ts":"1681736355972"} 2023-04-17 12:59:15,976 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:59:16,062 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:59:16,070 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, UNASSIGN}] 2023-04-17 12:59:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:16,074 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, UNASSIGN 2023-04-17 12:59:16,077 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=63889a78c730fa94931396cdfa19ec86, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:16,077 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736356076"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736356076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736356076"}]},"ts":"1681736356076"} 2023-04-17 12:59:16,090 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:59:16,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:16,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 63889a78c730fa94931396cdfa19ec86, disabling compactions & flushes 2023-04-17 12:59:16,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:16,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:16,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. after waiting 0 ms 2023-04-17 12:59:16,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:16,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 63889a78c730fa94931396cdfa19ec86 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=A 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=B 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63889a78c730fa94931396cdfa19ec86, store=C 2023-04-17 12:59:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ebce71e0cda244d8b1d5b486541ae066 2023-04-17 12:59:16,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/548d201c352d4be69129231604015f86 2023-04-17 12:59:16,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bb947955106643aab3c2104f8fd3f8fc 2023-04-17 12:59:16,787 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/A/ebce71e0cda244d8b1d5b486541ae066 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ebce71e0cda244d8b1d5b486541ae066 2023-04-17 12:59:16,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ebce71e0cda244d8b1d5b486541ae066, entries=150, sequenceid=557, filesize=11.9 K 2023-04-17 12:59:16,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/B/548d201c352d4be69129231604015f86 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/548d201c352d4be69129231604015f86 2023-04-17 12:59:16,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/548d201c352d4be69129231604015f86, entries=150, sequenceid=557, filesize=11.9 K 2023-04-17 12:59:16,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/.tmp/C/bb947955106643aab3c2104f8fd3f8fc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb947955106643aab3c2104f8fd3f8fc 2023-04-17 12:59:16,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb947955106643aab3c2104f8fd3f8fc, entries=150, sequenceid=557, filesize=11.9 K 2023-04-17 12:59:16,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 63889a78c730fa94931396cdfa19ec86 in 629ms, sequenceid=557, compaction requested=true 2023-04-17 12:59:16,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9dce0b93425d4991823f4642f9b563b1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/804ea78455824f5b96c541dcfb2f866b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/083e7f48e22b49dfab9666a269e44cd7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/b1eebb18ae5a4342a8c55843f33b8c9d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a661f92acb5e4073924675a8a0494803, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/64ec4402b350440c9e44c2838fbc532c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/1779b00a95ea4cd7ae9e2ec0f930fb43, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ba223147ec90453485a6a10b38262ab7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47] to archive 2023-04-17 12:59:16,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:16,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9dce0b93425d4991823f4642f9b563b1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9dce0b93425d4991823f4642f9b563b1 2023-04-17 12:59:16,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/692a002123c346e3beb0a5e37fc60438 2023-04-17 12:59:16,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/223daeb8a3374f22b3030d386ac47841 2023-04-17 12:59:16,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a184af147b0c4dae91b9640c020c5e64 2023-04-17 12:59:16,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/804ea78455824f5b96c541dcfb2f866b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/804ea78455824f5b96c541dcfb2f866b 2023-04-17 12:59:16,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a073a3a412114ea2924ef352a148d917 2023-04-17 12:59:16,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/638e7b11135f4ba1b542ec11bdfc24d8 2023-04-17 12:59:16,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/4fe5007c5bb748bf97614cc55114d06f 2023-04-17 12:59:16,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/083e7f48e22b49dfab9666a269e44cd7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/083e7f48e22b49dfab9666a269e44cd7 2023-04-17 12:59:16,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2dd1395a0842402ca3c7dbe5080dc391 2023-04-17 12:59:16,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f5b93e28ed49420bb8ac8be0e28b8dc5 2023-04-17 12:59:16,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/6201bb039dbc4ba5b5eb8583c94aa7e6 2023-04-17 12:59:17,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/b1eebb18ae5a4342a8c55843f33b8c9d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/b1eebb18ae5a4342a8c55843f33b8c9d 2023-04-17 12:59:17,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/81bac9a92cd94ca48afb7ac981e804db 2023-04-17 12:59:17,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/534b5a46c73d45b8862f39664995b274 2023-04-17 12:59:17,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a661f92acb5e4073924675a8a0494803 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/a661f92acb5e4073924675a8a0494803 2023-04-17 12:59:17,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f1f5d1765538475b9c11178b6fb97ec6 2023-04-17 12:59:17,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/275a2310ad984c49a36dbc70daffcee9 2023-04-17 12:59:17,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/64ec4402b350440c9e44c2838fbc532c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/64ec4402b350440c9e44c2838fbc532c 2023-04-17 12:59:17,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ca8904e26d0848ddae8780195123b4a5 2023-04-17 12:59:17,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/babca5d1524343f98f268ccc9913120a 2023-04-17 12:59:17,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/1779b00a95ea4cd7ae9e2ec0f930fb43 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/1779b00a95ea4cd7ae9e2ec0f930fb43 2023-04-17 12:59:17,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ade875c52a5748ffba3107517f6047a0 2023-04-17 12:59:17,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/2c8c2e945b3a4898b85868dd3a2f5584 2023-04-17 12:59:17,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ba223147ec90453485a6a10b38262ab7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ba223147ec90453485a6a10b38262ab7 2023-04-17 12:59:17,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/35e36893df69480ca9f072e687c4b57d 2023-04-17 12:59:17,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:17,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/9d7494fb30d145409ba387a145a2b764 2023-04-17 12:59:17,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/076d66faafcc41708dc9ae292b48dc47 2023-04-17 12:59:17,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7f433039f1ad40a0b51dee9c1ff6d5ab, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/670fef4af75a403cb884ae240fe89033, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/f1cf17f15d5049609fd0d21e8ff389ce, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b47c9b62bc6f4e8292a45861a0a66541, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ee4b3ae343347d98a50e3978746ac2e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6110ba6f5427416789eb7ffc47abda9a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/2e6368e9f5004138ba14ab5918e833eb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/99cfa0f59fd94e43b1a8cdd3d17353d1, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729] to archive 2023-04-17 12:59:17,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:17,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7f433039f1ad40a0b51dee9c1ff6d5ab to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7f433039f1ad40a0b51dee9c1ff6d5ab 2023-04-17 12:59:17,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/17a0c150ceeb4f5d9afb45fa47a3b05d 2023-04-17 12:59:17,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9b6dcf2d4d6243119656a1632b4d2349 2023-04-17 12:59:17,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/670fef4af75a403cb884ae240fe89033 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/670fef4af75a403cb884ae240fe89033 2023-04-17 12:59:17,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/28aa234676824e80a66ced10d9de9a33 2023-04-17 12:59:17,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/acaa4ad65af94303bb5e6cce0fc5893b 2023-04-17 12:59:17,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/d138ff62bc82488c81faac786be1b722 2023-04-17 12:59:17,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/f1cf17f15d5049609fd0d21e8ff389ce to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/f1cf17f15d5049609fd0d21e8ff389ce 2023-04-17 12:59:17,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b03c4f3e05cb4ba88769ddfee8df872d 2023-04-17 12:59:17,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/787fc1f24cac45c29946495f8ecf941f 2023-04-17 12:59:17,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/873a64b28e1f46b998cffa7275835bd3 2023-04-17 12:59:17,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b47c9b62bc6f4e8292a45861a0a66541 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b47c9b62bc6f4e8292a45861a0a66541 2023-04-17 12:59:17,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/950b2bb7ee9d42789e1854afd372f2fb 2023-04-17 12:59:17,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/454bed70e4a8479fbb0a6d5dbe50d43c 2023-04-17 12:59:17,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ee4b3ae343347d98a50e3978746ac2e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/3ee4b3ae343347d98a50e3978746ac2e 2023-04-17 12:59:17,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/8d300ddb249441cd9164d589d63fc8d3 2023-04-17 12:59:17,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bf5f7bdc22824581bf4ba00193443931 2023-04-17 12:59:17,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6110ba6f5427416789eb7ffc47abda9a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6110ba6f5427416789eb7ffc47abda9a 2023-04-17 12:59:17,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/907643caa3354075ad3ddb3b02af3a6b 2023-04-17 12:59:17,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/95ad34f703f3461fa7ccfd19312709d8 2023-04-17 12:59:17,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/2e6368e9f5004138ba14ab5918e833eb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/2e6368e9f5004138ba14ab5918e833eb 2023-04-17 12:59:17,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/9c6022535dc745f9b8a78709e62cbfe1 2023-04-17 12:59:17,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/45b05eb7aef942b0b0fc96019aa75dbc 2023-04-17 12:59:17,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/99cfa0f59fd94e43b1a8cdd3d17353d1 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/99cfa0f59fd94e43b1a8cdd3d17353d1 2023-04-17 12:59:17,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/7d48c5261c084848817eaacfa6dd8955 2023-04-17 12:59:17,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/ae0adfa9abec4957bd38e04df9b67578 2023-04-17 12:59:17,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/b85bf2bc608a427db4551f86ae20bd4e 2023-04-17 12:59:17,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/6cd1513f95774ce18fa079d401da6729 2023-04-17 12:59:17,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f0b6029ddc0e4e29960727a2ce19879e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4d7827e307254aee98a0793d0c6ffd29, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/d786526d078b4677a11c5b6eb98eadfe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/a3e12072c0c04437a5d0572569310cf9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e447a9eff47d4956b6e1c6650e5e7e15, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/419b264065dc42cf8bcff32559cdc64b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/13acebe3b372429eb1fc6259eaa785cb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/ffb7f5a254b24a7987f5b7671bda01ba, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e] to archive 2023-04-17 12:59:17,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:17,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f0b6029ddc0e4e29960727a2ce19879e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f0b6029ddc0e4e29960727a2ce19879e 2023-04-17 12:59:17,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/f8803797cf0944e88521ee66ae0f6ee6 2023-04-17 12:59:17,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7c689077652e4ea5b57f2ab8ce8ad907 2023-04-17 12:59:17,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4d7827e307254aee98a0793d0c6ffd29 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4d7827e307254aee98a0793d0c6ffd29 2023-04-17 12:59:17,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/47bd0d781e47406fa88a6866f6b9ebfa 2023-04-17 12:59:17,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/46e60d38cd954816832ac73b0d0d1eb8 2023-04-17 12:59:17,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/c5a086b2aaa044c6af9a5c5de1f277b4 2023-04-17 12:59:17,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/d786526d078b4677a11c5b6eb98eadfe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/d786526d078b4677a11c5b6eb98eadfe 2023-04-17 12:59:17,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/95119c7ad871441a92bff1e662167321 2023-04-17 12:59:17,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b0d55d9e873844e587673c52a73c6048 2023-04-17 12:59:17,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb1420fbc010451e8d56d36bd8a3af0f 2023-04-17 12:59:17,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/a3e12072c0c04437a5d0572569310cf9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/a3e12072c0c04437a5d0572569310cf9 2023-04-17 12:59:17,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/411cedeb688949f0842dfc312286c2a3 2023-04-17 12:59:17,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/9da8caffc2c34eb3a4686e3afc23e532 2023-04-17 12:59:17,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e447a9eff47d4956b6e1c6650e5e7e15 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e447a9eff47d4956b6e1c6650e5e7e15 2023-04-17 12:59:17,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5c7589b2bdf8456ba1e9d22d434911df 2023-04-17 12:59:17,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4ec38121e3d743189b8bffb9049763b3 2023-04-17 12:59:17,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/419b264065dc42cf8bcff32559cdc64b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/419b264065dc42cf8bcff32559cdc64b 2023-04-17 12:59:17,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bd6765bdf94e4891b0117e4cdf146572 2023-04-17 12:59:17,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/5b8665f320e44b608a0707e3f3ccd9b9 2023-04-17 12:59:17,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/13acebe3b372429eb1fc6259eaa785cb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/13acebe3b372429eb1fc6259eaa785cb 2023-04-17 12:59:17,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/60a35daae5364bbe80967799da5948fb 2023-04-17 12:59:17,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/b3709806892646d9a0748cc4e3c9f91c 2023-04-17 12:59:17,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/ffb7f5a254b24a7987f5b7671bda01ba to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/ffb7f5a254b24a7987f5b7671bda01ba 2023-04-17 12:59:17,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/7aa85e4da5ae4966ad9cdb44b4baabe7 2023-04-17 12:59:17,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/e41c9307858844a290330b2fb7ac025f 2023-04-17 12:59:17,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/02c72b4afca2430092d319f944118a06 2023-04-17 12:59:17,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/603629ad92d446f890ae3e1671a10f6e 2023-04-17 12:59:17,229 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/recovered.edits/560.seqid, newMaxSeqId=560, maxSeqId=1 2023-04-17 12:59:17,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86. 2023-04-17 12:59:17,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 63889a78c730fa94931396cdfa19ec86: 2023-04-17 12:59:17,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:17,233 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=63889a78c730fa94931396cdfa19ec86, regionState=CLOSED 2023-04-17 12:59:17,233 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736357233"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736357233"}]},"ts":"1681736357233"} 2023-04-17 12:59:17,236 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-17 12:59:17,236 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 63889a78c730fa94931396cdfa19ec86, server=jenkins-hbase17.apache.org,40215,1681736213280 in 1.1440 sec 2023-04-17 12:59:17,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-17 12:59:17,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63889a78c730fa94931396cdfa19ec86, UNASSIGN in 1.1660 sec 2023-04-17 12:59:17,238 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736357238"}]},"ts":"1681736357238"} 2023-04-17 12:59:17,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:59:17,240 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:59:17,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2800 sec 2023-04-17 12:59:18,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 12:59:18,084 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-17 12:59:18,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:59:18,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,090 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,091 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 12:59:18,099 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:18,105 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/recovered.edits] 2023-04-17 12:59:18,111 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ebce71e0cda244d8b1d5b486541ae066 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/ebce71e0cda244d8b1d5b486541ae066 2023-04-17 12:59:18,123 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f2fb9d7a44f441f4bcde0cdf1fe71369 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f2fb9d7a44f441f4bcde0cdf1fe71369 2023-04-17 12:59:18,133 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f738b8d62db4473e8322073ea58c7f89 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/A/f738b8d62db4473e8322073ea58c7f89 2023-04-17 12:59:18,139 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/548d201c352d4be69129231604015f86 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/548d201c352d4be69129231604015f86 2023-04-17 12:59:18,142 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/779ee4b284b44253b924531832ab6c61 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/779ee4b284b44253b924531832ab6c61 2023-04-17 12:59:18,147 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bad04f08edc7477aaba0723d0cc585cd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/B/bad04f08edc7477aaba0723d0cc585cd 2023-04-17 12:59:18,156 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3e9f830b958a41fb90c32b1e0d7a3c70 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/3e9f830b958a41fb90c32b1e0d7a3c70 2023-04-17 12:59:18,159 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4e067cfa1a914389971566c27115cefe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/4e067cfa1a914389971566c27115cefe 2023-04-17 12:59:18,163 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb947955106643aab3c2104f8fd3f8fc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/C/bb947955106643aab3c2104f8fd3f8fc 2023-04-17 12:59:18,191 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/recovered.edits/560.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86/recovered.edits/560.seqid 2023-04-17 12:59:18,191 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/63889a78c730fa94931396cdfa19ec86 2023-04-17 12:59:18,191 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:59:18,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 12:59:18,202 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,208 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:59:18,219 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:59:18,223 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,223 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:59:18,223 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736358223"}]},"ts":"9223372036854775807"} 2023-04-17 12:59:18,246 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:59:18,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 63889a78c730fa94931396cdfa19ec86, NAME => 'TestAcidGuarantees,,1681736333183.63889a78c730fa94931396cdfa19ec86.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:59:18,246 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:59:18,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736358246"}]},"ts":"9223372036854775807"} 2023-04-17 12:59:18,255 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:59:18,265 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 181 msec 2023-04-17 12:59:18,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 12:59:18,397 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-17 12:59:18,411 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=593 (was 530) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1113 (was 971) - SystemLoadAverage LEAK? -, ProcessCount=195 (was 195), AvailableMemoryMB=254 (was 252) - AvailableMemoryMB LEAK? - 2023-04-17 12:59:18,425 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=595, MaxFileDescriptor=60000, SystemLoadAverage=1113, ProcessCount=195, AvailableMemoryMB=254 2023-04-17 12:59:18,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:59:18,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$4(2112): Client=jenkins//136.243.18.41 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:59:18,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:18,431 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 12:59:18,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(697): Client=jenkins//136.243.18.41 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-17 12:59:18,431 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 12:59:18,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 12:59:18,432 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,433 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 empty. 2023-04-17 12:59:18,434 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,434 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:59:18,455 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:59:18,457 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4c484ccfa7891407c1464a9d303bfd85, NAME => 'TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4c484ccfa7891407c1464a9d303bfd85, disabling compactions & flushes 2023-04-17 12:59:18,482 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. after waiting 0 ms 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,482 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,482 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:18,489 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 12:59:18,490 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736358490"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736358490"}]},"ts":"1681736358490"} 2023-04-17 12:59:18,491 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 12:59:18,492 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 12:59:18,492 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736358492"}]},"ts":"1681736358492"} 2023-04-17 12:59:18,493 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 12:59:18,495 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, ASSIGN}] 2023-04-17 12:59:18,497 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, ASSIGN 2023-04-17 12:59:18,498 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, ASSIGN; state=OFFLINE, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=false 2023-04-17 12:59:18,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 12:59:18,649 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:18,649 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736358649"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736358649"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736358649"}]},"ts":"1681736358649"} 2023-04-17 12:59:18,651 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:59:18,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 12:59:18,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c484ccfa7891407c1464a9d303bfd85, NAME => 'TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:59:18,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:59:18,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,807 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,808 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A 2023-04-17 12:59:18,809 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A 2023-04-17 12:59:18,809 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:18,809 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName A 2023-04-17 12:59:18,810 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:18,810 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,813 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B 2023-04-17 12:59:18,813 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B 2023-04-17 12:59:18,813 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:18,813 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName B 2023-04-17 12:59:18,814 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:18,814 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,818 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C 2023-04-17 12:59:18,818 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C 2023-04-17 12:59:18,818 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:18,818 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName C 2023-04-17 12:59:18,819 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:18,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:59:18,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:18,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 12:59:18,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 4c484ccfa7891407c1464a9d303bfd85; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71894567, jitterRate=0.07131253182888031}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:59:18,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:18,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., pid=56, masterSystemTime=1681736358802 2023-04-17 12:59:18,831 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:18,832 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736358831"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736358831"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736358831"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736358831"}]},"ts":"1681736358831"} 2023-04-17 12:59:18,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:18,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-17 12:59:18,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 in 182 msec 2023-04-17 12:59:18,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-17 12:59:18,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, ASSIGN in 339 msec 2023-04-17 12:59:18,836 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 12:59:18,836 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736358836"}]},"ts":"1681736358836"} 2023-04-17 12:59:18,837 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 12:59:18,838 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 12:59:18,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 411 msec 2023-04-17 12:59:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 12:59:19,035 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-17 12:59:19,036 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x0c217430 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:19,076 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ae8f5b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:19,079 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:19,080 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:19,090 DEBUG [Listener at localhost.localdomain/43083] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 12:59:19,094 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:56674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 12:59:19,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 12:59:19,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$12(2454): Client=jenkins//136.243.18.41 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 12:59:19,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:19,182 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:59:19,194 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 12:59:19,194 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 12:59:19,222 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 12:59:19,238 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, REOPEN/MOVE}] 2023-04-17 12:59:19,242 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, REOPEN/MOVE 2023-04-17 12:59:19,245 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:19,245 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736359245"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736359245"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736359245"}]},"ts":"1681736359245"} 2023-04-17 12:59:19,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:59:19,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 4c484ccfa7891407c1464a9d303bfd85, disabling compactions & flushes 2023-04-17 12:59:19,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. after waiting 0 ms 2023-04-17 12:59:19,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 12:59:19,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:19,442 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4c484ccfa7891407c1464a9d303bfd85 to self. 2023-04-17 12:59:19,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,458 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=CLOSED 2023-04-17 12:59:19,458 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736359458"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736359458"}]},"ts":"1681736359458"} 2023-04-17 12:59:19,475 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-17 12:59:19,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 in 216 msec 2023-04-17 12:59:19,482 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase17.apache.org,40215,1681736213280; forceNewPlan=false, retain=true 2023-04-17 12:59:19,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=OPENING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:19,633 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736359633"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736359633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736359633"}]},"ts":"1681736359633"} 2023-04-17 12:59:19,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:59:19,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c484ccfa7891407c1464a9d303bfd85, NAME => 'TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.', STARTKEY => '', ENDKEY => ''} 2023-04-17 12:59:19,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 12:59:19,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7894): checking encryption for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(7897): checking classloading for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,796 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,810 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A 2023-04-17 12:59:19,810 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A 2023-04-17 12:59:19,811 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:19,811 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName A 2023-04-17 12:59:19,824 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:19,825 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,830 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B 2023-04-17 12:59:19,831 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B 2023-04-17 12:59:19,831 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:19,831 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName B 2023-04-17 12:59:19,834 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:19,834 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,835 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C 2023-04-17 12:59:19,835 DEBUG [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C 2023-04-17 12:59:19,835 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 12:59:19,836 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c484ccfa7891407c1464a9d303bfd85 columnFamilyName C 2023-04-17 12:59:19,842 INFO [StoreOpener-4c484ccfa7891407c1464a9d303bfd85-1] regionserver.HStore(310): Store=4c484ccfa7891407c1464a9d303bfd85/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 12:59:19,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 12:59:19,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1055): writing seq id for 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:19,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1072): Opened 4c484ccfa7891407c1464a9d303bfd85; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59939501, jitterRate=-0.10683183372020721}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 12:59:19,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(965): Region open journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:19,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., pid=61, masterSystemTime=1681736359790 2023-04-17 12:59:19,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase17:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:19,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:19,851 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736359851"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681736359851"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681736359851"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681736359851"}]},"ts":"1681736359851"} 2023-04-17 12:59:19,854 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-17 12:59:19,855 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 in 217 msec 2023-04-17 12:59:19,857 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-17 12:59:19,857 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, REOPEN/MOVE in 617 msec 2023-04-17 12:59:19,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-17 12:59:19,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 636 msec 2023-04-17 12:59:19,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 764 msec 2023-04-17 12:59:19,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-17 12:59:19,864 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5c51ec to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:19,910 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ddb3e63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:19,910 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x5be55982 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:19,953 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@630ea023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:19,954 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x2f11e84a to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:19,982 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec98d63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:19,983 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3a1010a6 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,008 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ae3ec16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,009 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x4143b9ae to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,082 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bec0400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,083 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x34b8bf14 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,139 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f6f7187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,139 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x387217bb to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,198 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c34fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,199 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7c00a18c to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,245 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5467be78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,245 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x3da68fe0 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,303 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44535519, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,304 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(139): Connect 0x7fd90ac8 to 127.0.0.1:62647 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 12:59:20,391 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7689435c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 12:59:20,494 DEBUG [hconnection-0x2f0f22a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,511 DEBUG [hconnection-0x6c6d30dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,513 DEBUG [hconnection-0x799508ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,515 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,523 DEBUG [hconnection-0x6c84185e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,526 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,531 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,532 DEBUG [hconnection-0x5d31b582-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,535 DEBUG [hconnection-0x4b9294a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:20,550 DEBUG [hconnection-0x2e2b85c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,550 DEBUG [hconnection-0x6548ffa5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,569 DEBUG [hconnection-0x37c5c363-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,570 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,571 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,572 DEBUG [hconnection-0x1082853b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 12:59:20,618 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,619 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,624 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,625 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:20,634 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:20,634 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:20,634 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:20,634 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:20,651 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:20,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:20,652 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 136.243.18.41:48198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 12:59:20,652 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:20,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:20,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:20,653 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:20,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:20,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:20,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:20,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:20,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:20,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:20,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:20,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:20,655 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:20,659 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:20,660 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:20,678 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 12:59:20,695 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:20,695 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:20,695 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:20,695 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:20,696 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:20,696 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:20,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736360744, max timestamp=1681736360786Num uniques 859; 2023-04-17 12:59:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736360744, max timestamp=1681736360786Num uniques 859; 2023-04-17 12:59:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736360744, max timestamp=1681736360786Num uniques 859; 2023-04-17 12:59:20,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:20,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:20,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:20,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736420843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:20,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736420849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1681736360804, max timestamp=1681736360844Num uniques 1018; 2023-04-17 12:59:20,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=7681, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:59:20,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681736360804, max timestamp=1681736360844Num uniques 1018; 2023-04-17 12:59:20,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:59:20,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1681736360804, max timestamp=1681736360844Num uniques 1018; 2023-04-17 12:59:20,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=7681, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 12:59:20,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681736360883, max timestamp=1681736360947Num uniques 1027; 2023-04-17 12:59:20,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 12:59:20,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681736360883, max timestamp=1681736360947Num uniques 1027; 2023-04-17 12:59:20,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 12:59:20,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681736360883, max timestamp=1681736360947Num uniques 1027; 2023-04-17 12:59:20,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-17 12:59:20,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:20,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736420986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:20,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736420988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:20,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:20,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736420994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736421001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736421001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736421098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736421100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736421103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736421104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736421104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,244 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417937015f921124f81a4158ef76155d2b8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417937015f921124f81a4158ef76155d2b8 2023-04-17 12:59:21,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736421306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736421315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736421315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736421318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736421319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736421621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736421627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736421627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736421629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736421629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:21,749 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/e239a727021c407a9acabe0c7cb3e466 2023-04-17 12:59:21,869 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/35ee474a8b704e68a90dbcae3c804c12 2023-04-17 12:59:21,975 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d5df98691e4e40518b10ec06acda04a0 2023-04-17 12:59:22,003 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/e239a727021c407a9acabe0c7cb3e466 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466 2023-04-17 12:59:22,034 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466, entries=100, sequenceid=13, filesize=18.4 K 2023-04-17 12:59:22,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/35ee474a8b704e68a90dbcae3c804c12 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12 2023-04-17 12:59:22,129 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 12:59:22,140 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d5df98691e4e40518b10ec06acda04a0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0 2023-04-17 12:59:22,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736422130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736422131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736422138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736422142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736422146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,191 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 12:59:22,194 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=145.36 KB/148850 for 4c484ccfa7891407c1464a9d303bfd85 in 1516ms, sequenceid=13, compaction requested=false 2023-04-17 12:59:22,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:22,194 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:22,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:22,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:22,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:22,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:22,214 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:22,216 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:22,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67d3ee0a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:22,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:22,216 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,226 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:22,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:22,248 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:22,248 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:22,248 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:22,248 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:22,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:22,276 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:22,276 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:22,276 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:22,276 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:22,276 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:22,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,283 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:22,284 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:22,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:22,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:22,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:22,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:22,298 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:22,298 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:22,299 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:22,299 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:22,299 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:22,399 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:22,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:22,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:22,450 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:22,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:22,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:22,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:22,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,451 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:22,458 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:22,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:22,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:22,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:22,460 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,460 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:22,460 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:22,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:22,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:22,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:22,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:22,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:22,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:22,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:22,478 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:22,478 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:22,479 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=145.36 KB heapSize=408.69 KB 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:22,479 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:22,615 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f6b21f64827c49a2a9c510217304070f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f6b21f64827c49a2a9c510217304070f 2023-04-17 12:59:22,675 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/1e4f73e6cf6041a3840cddab7223902e 2023-04-17 12:59:22,775 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 12:59:23,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:23,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:23,180 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/9e983480b9dd4a0d9a7a497097e3d6a4 2023-04-17 12:59:23,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736423184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736423185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736423185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736423194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736423201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736423302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736423302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736423303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736423309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736423310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736423509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736423510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736423510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736423519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736423520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,668 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d5c8d32aaba64b74a35940264253039b 2023-04-17 12:59:23,673 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/1e4f73e6cf6041a3840cddab7223902e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e 2023-04-17 12:59:23,692 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e, entries=150, sequenceid=84, filesize=25.1 K 2023-04-17 12:59:23,693 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/9e983480b9dd4a0d9a7a497097e3d6a4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4 2023-04-17 12:59:23,712 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4, entries=150, sequenceid=84, filesize=11.6 K 2023-04-17 12:59:23,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d5c8d32aaba64b74a35940264253039b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b 2023-04-17 12:59:23,774 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b, entries=150, sequenceid=84, filesize=11.6 K 2023-04-17 12:59:23,785 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~408.64 KB/418448, currentSize=40.25 KB/41220 for 4c484ccfa7891407c1464a9d303bfd85 in 1306ms, sequenceid=84, compaction requested=false 2023-04-17 12:59:23,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:23,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:23,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:23,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:23,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:23,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:23,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:23,791 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:23,792 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,792 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:23,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:23,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e1b86bc[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:23,793 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:23,793 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:23,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:23,796 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:23,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:23,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:23,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:23,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:23,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:23,799 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:23,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:23,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:23,799 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:23,799 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:23,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:23,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,802 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:23,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:23,826 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,830 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:23,830 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:23,857 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:23,857 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:23,857 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:23,857 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:23,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:23,857 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:23,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:23,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:23,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:23,861 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:23,862 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:23,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736363895, max timestamp=1681736363905Num uniques 859; 2023-04-17 12:59:23,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:23,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736363895, max timestamp=1681736363905Num uniques 859; 2023-04-17 12:59:23,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:23,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681736363895, max timestamp=1681736363905Num uniques 859; 2023-04-17 12:59:23,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 12:59:23,939 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417273d5cea919e4714a19fb32346c3ad1d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417273d5cea919e4714a19fb32346c3ad1d 2023-04-17 12:59:23,963 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:23,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:23,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736423977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736423978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736423979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:23,988 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=94, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/46a5956262374967bdc2d81bd687f0f7 2023-04-17 12:59:23,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:23,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:23,995 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:23,995 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:23,995 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:23,995 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:24,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736423986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736423987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,001 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,015 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:24,021 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,021 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:24,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:24,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:24,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:24,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,030 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:24,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:24,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:24,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:24,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:24,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:24,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:24,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,042 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:24,042 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:24,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,054 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,055 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,055 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:24,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ca7ed[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:24,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:24,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,074 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,075 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:24,075 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:24,075 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:24,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:24,082 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,082 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:24,082 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,083 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:24,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736424090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736424094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736424096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736424102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736424107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:24,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:24,115 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:24,115 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:24,115 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:24,115 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:24,115 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:24,123 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,126 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,127 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,127 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,128 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,144 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:24,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:24,144 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:24,144 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:24,144 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,244 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:24,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:24,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:24,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:24,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:24,278 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:24,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:24,278 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:24,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,294 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,296 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:24,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:24,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:24,299 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:24,299 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,299 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:24,301 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:24,301 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:24,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,302 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:24,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,306 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,307 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:24,307 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:24,307 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736424301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,307 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:24,307 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:24,307 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,307 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:24,307 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:24,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736424306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736424310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,313 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736424310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,314 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,315 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:24,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736424318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbbef36[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:24,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:24,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,414 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,415 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:24,415 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:24,415 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:24,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:24,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,422 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:24,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:24,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:24,432 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:24,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:24,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:24,433 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:24,433 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:24,438 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,438 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,439 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,470 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:24,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:24,492 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,492 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,493 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:24,493 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/639c1319876a481eb576ee2b7c7a3419 2023-04-17 12:59:24,593 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:24,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:24,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:24,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736424609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736424611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736424617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736424617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:24,622 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:24,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:24,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:24,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,623 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:24,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736424638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:24,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:24,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:24,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:24,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:24,645 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:24,647 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,647 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:24,647 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:24,647 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:24,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:24,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:24,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:24,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/3d9d6e48a6444e31a171ee47ee08e565 2023-04-17 12:59:24,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,649 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:24,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e0f535[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:24,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:24,652 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,652 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,653 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:24,653 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:24,653 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:24,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:24,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,654 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:24,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:24,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:24,654 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:24,654 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:24,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:24,655 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:24,655 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:24,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:24,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,656 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,657 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/46a5956262374967bdc2d81bd687f0f7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7 2023-04-17 12:59:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:24,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7, entries=150, sequenceid=94, filesize=25.1 K 2023-04-17 12:59:24,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/639c1319876a481eb576ee2b7c7a3419 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419 2023-04-17 12:59:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:24,710 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:24,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:24,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:24,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:24,710 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:24,711 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:24,710 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:24,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419, entries=150, sequenceid=94, filesize=11.6 K 2023-04-17 12:59:24,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/3d9d6e48a6444e31a171ee47ee08e565 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565 2023-04-17 12:59:24,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565, entries=150, sequenceid=94, filesize=11.6 K 2023-04-17 12:59:24,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for 4c484ccfa7891407c1464a9d303bfd85 in 899ms, sequenceid=94, compaction requested=true 2023-04-17 12:59:24,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:24,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:24,729 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:24,729 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:24,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:24,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:24,733 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 70290 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:24,733 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:24,733 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,733 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=68.6 K 2023-04-17 12:59:24,734 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:24,734 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:24,734 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,734 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=32.5 K 2023-04-17 12:59:24,734 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting e239a727021c407a9acabe0c7cb3e466, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681736360659 2023-04-17 12:59:24,735 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 35ee474a8b704e68a90dbcae3c804c12, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681736360659 2023-04-17 12:59:24,735 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 9e983480b9dd4a0d9a7a497097e3d6a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681736360980 2023-04-17 12:59:24,735 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 639c1319876a481eb576ee2b7c7a3419, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:24,736 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1e4f73e6cf6041a3840cddab7223902e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681736360980 2023-04-17 12:59:24,736 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 46a5956262374967bdc2d81bd687f0f7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:24,747 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#453 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:24,790 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#454 average throughput is 0.68 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:24,811 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:24,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:24,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:24,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:24,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:24,834 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:24,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:24,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:24,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:24,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:24,844 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,849 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:24,849 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:24,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:24,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:24,852 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:24,852 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:24,852 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:24,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:24,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:24,862 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,862 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:24,862 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-17 12:59:24,865 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:24,866 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:24,866 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:24,866 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:24,866 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:24,866 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:24,942 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/ca1cf3a871834c2a9944a9060b609d87 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ca1cf3a871834c2a9944a9060b609d87 2023-04-17 12:59:24,974 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into ca1cf3a871834c2a9944a9060b609d87(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:24,974 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:24,974 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=13, startTime=1681736364729; duration=0sec 2023-04-17 12:59:24,974 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:24,974 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:24,990 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:24,990 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:24,990 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:24,990 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=32.5 K 2023-04-17 12:59:24,990 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d5df98691e4e40518b10ec06acda04a0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681736360659 2023-04-17 12:59:24,991 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d5c8d32aaba64b74a35940264253039b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681736360980 2023-04-17 12:59:24,992 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3d9d6e48a6444e31a171ee47ee08e565, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:24,999 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:25,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:25,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736425148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736425151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736425154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736425158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736425159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,246 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/6a3f29180d81417b9e67bb63bc7b5d80 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/6a3f29180d81417b9e67bb63bc7b5d80 2023-04-17 12:59:25,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736425261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736425265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736425266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736425266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,278 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into 6a3f29180d81417b9e67bb63bc7b5d80(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:25,279 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:25,279 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=13, startTime=1681736364729; duration=0sec 2023-04-17 12:59:25,279 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:25,361 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e6f45abeac2c4a88b3a5d9b1d2395c30 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e6f45abeac2c4a88b3a5d9b1d2395c30 2023-04-17 12:59:25,447 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/b17b63262a6844129a03ee7544fc38bd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/b17b63262a6844129a03ee7544fc38bd 2023-04-17 12:59:25,453 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into b17b63262a6844129a03ee7544fc38bd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:25,453 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:25,453 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=13, startTime=1681736364729; duration=0sec 2023-04-17 12:59:25,453 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:25,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736425470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736425470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736425474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736425483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736425778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736425787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736425789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736425794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:25,799 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/a73f0152e4884e85bdfcdacbfe1fb763 2023-04-17 12:59:25,882 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/7aff95a8ff594e30921e6f838da17050 2023-04-17 12:59:25,950 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/dc71f7c161a94a6e8cf2a3d8c03f617b 2023-04-17 12:59:25,967 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/a73f0152e4884e85bdfcdacbfe1fb763 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763 2023-04-17 12:59:25,993 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763, entries=150, sequenceid=131, filesize=25.2 K 2023-04-17 12:59:25,994 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/7aff95a8ff594e30921e6f838da17050 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050 2023-04-17 12:59:26,011 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050, entries=150, sequenceid=131, filesize=11.7 K 2023-04-17 12:59:26,012 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/dc71f7c161a94a6e8cf2a3d8c03f617b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b 2023-04-17 12:59:26,022 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b, entries=150, sequenceid=131, filesize=11.7 K 2023-04-17 12:59:26,023 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=53.67 KB/54960 for 4c484ccfa7891407c1464a9d303bfd85 in 1161ms, sequenceid=131, compaction requested=false 2023-04-17 12:59:26,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:26,023 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,024 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,033 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fca553[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,035 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,035 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,037 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,037 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,040 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:26,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,044 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,044 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,044 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,044 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,044 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,046 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,046 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,046 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,047 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,049 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,051 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:26,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:26,052 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:26,052 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:26,051 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,052 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,052 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,052 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,152 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:26,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:26,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:26,170 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:26,170 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:26,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:26,174 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:26,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:26,174 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:26,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,178 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,180 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,180 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,182 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:26,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:26,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:26,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:26,194 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,195 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:26,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:26,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:26,198 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,198 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:26,198 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:26,198 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,198 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,205 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,206 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,207 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,207 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,207 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,207 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@135b378e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,209 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,211 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,211 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,211 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,211 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,211 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,211 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,213 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,213 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,213 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,214 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:26,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,219 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,219 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,219 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,219 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,219 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,219 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f8498692c184455a8c8cdcae66e6e53a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8498692c184455a8c8cdcae66e6e53a 2023-04-17 12:59:26,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,221 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,223 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,223 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,225 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,228 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,230 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,230 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:26,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:26,230 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:26,230 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:26,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:26,251 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/c5289365994e49919b8b6da96d7d5725 2023-04-17 12:59:26,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736426261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736426286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736426294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736426295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/b33d3dfb83ad44c9b7966f5f73256142 2023-04-17 12:59:26,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736426301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,330 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:26,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:26,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:26,360 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:26,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:26,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:26,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/1703fcc53af14d10ba5132989194968a 2023-04-17 12:59:26,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,362 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,363 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:26,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:26,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/c5289365994e49919b8b6da96d7d5725 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725 2023-04-17 12:59:26,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:26,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:26,369 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:26,369 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,369 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:26,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736426366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:26,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:26,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725, entries=150, sequenceid=146, filesize=25.3 K 2023-04-17 12:59:26,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/b33d3dfb83ad44c9b7966f5f73256142 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142 2023-04-17 12:59:26,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:26,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:26,374 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,374 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,376 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,377 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,377 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,377 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717ba88c[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,382 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,383 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142, entries=150, sequenceid=146, filesize=11.8 K 2023-04-17 12:59:26,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/1703fcc53af14d10ba5132989194968a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a 2023-04-17 12:59:26,388 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a, entries=150, sequenceid=146, filesize=11.8 K 2023-04-17 12:59:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:26,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,392 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,392 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4c484ccfa7891407c1464a9d303bfd85 in 223ms, sequenceid=146, compaction requested=true 2023-04-17 12:59:26,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:26,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:26,393 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:26,393 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:26,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:26,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:26,394 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36011 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:26,394 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:26,394 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,394 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,394 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77591 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:26,394 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:26,394 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,394 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ca1cf3a871834c2a9944a9060b609d87, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=75.8 K 2023-04-17 12:59:26,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,394 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,394 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/6a3f29180d81417b9e67bb63bc7b5d80, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=35.2 K 2023-04-17 12:59:26,395 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ca1cf3a871834c2a9944a9060b609d87, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:26,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,395 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,396 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6a3f29180d81417b9e67bb63bc7b5d80, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:26,396 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a73f0152e4884e85bdfcdacbfe1fb763, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736363974 2023-04-17 12:59:26,397 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7aff95a8ff594e30921e6f838da17050, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736363974 2023-04-17 12:59:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,397 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c5289365994e49919b8b6da96d7d5725, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:26,397 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b33d3dfb83ad44c9b7966f5f73256142, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,402 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,415 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:26,423 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,423 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,423 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,423 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,424 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,424 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,424 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,424 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,424 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,426 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:26,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:26,426 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:26,426 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:26,435 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#462 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:26,508 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/d686a6c2487a4831b90f4a250497f9bd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/d686a6c2487a4831b90f4a250497f9bd 2023-04-17 12:59:26,512 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into d686a6c2487a4831b90f4a250497f9bd(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:26,512 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:26,512 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=13, startTime=1681736366393; duration=0sec 2023-04-17 12:59:26,513 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:26,513 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:26,514 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36011 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:26,514 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:26,514 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,514 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/b17b63262a6844129a03ee7544fc38bd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=35.2 K 2023-04-17 12:59:26,514 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b17b63262a6844129a03ee7544fc38bd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1681736363182 2023-04-17 12:59:26,515 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting dc71f7c161a94a6e8cf2a3d8c03f617b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681736363974 2023-04-17 12:59:26,515 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 1703fcc53af14d10ba5132989194968a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:26,526 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:26,531 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#464 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:26,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:26,552 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:26,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,552 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:26,553 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:26,553 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,553 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:26,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:26,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:26,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:26,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:26,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:26,573 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:26,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:26,575 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/5d6627628295453db50889c438d489c4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/5d6627628295453db50889c438d489c4 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:26,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:26,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:26,577 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,577 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,581 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,581 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f0a2cb[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,587 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,588 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,588 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into 5d6627628295453db50889c438d489c4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:26,588 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,589 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:26,589 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=13, startTime=1681736366393; duration=0sec 2023-04-17 12:59:26,589 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:26,588 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,589 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,589 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:26,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,597 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,597 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,597 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,597 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,597 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,599 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,603 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,608 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:26,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,608 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,608 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:26,608 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:26,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:26,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736426617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,708 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:26,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:26,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:26,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:26,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:26,718 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:26,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:26,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:26,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,718 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,719 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,719 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,720 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:26,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:26,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:26,721 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:26,721 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,721 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:26,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:26,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:26,722 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,722 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,724 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,724 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,724 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,724 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21344fdb[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,728 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,729 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,729 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,730 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736426730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,734 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,737 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,737 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,750 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,750 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,750 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,774 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:26,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:26,790 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,794 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,794 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:26,794 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:26,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,790 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,795 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,895 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:26,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:26,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:26,918 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:26,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:26,918 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:26,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,920 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,921 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:26,922 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:26,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:26,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:26,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:26,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:26,939 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:26,939 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:26,939 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:26,944 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:26,945 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,945 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:26,945 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:26,945 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:26,945 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:26,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:26,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736426942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,947 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,947 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:26,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:26,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b19eb34[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:26,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:26,963 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:26,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:26,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:26,970 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:26,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:26,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:26,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:26,980 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:26,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:26,981 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:26,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:26,981 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:26,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:26,982 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:26,986 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,986 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:26,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,986 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:26,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:26,998 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,019 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,036 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:27,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,037 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/fcaeccb45c3d43109745ae652c301937 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/fcaeccb45c3d43109745ae652c301937 2023-04-17 12:59:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,041 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178daa27fd2bb444daaead53de69a11201 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178daa27fd2bb444daaead53de69a11201 2023-04-17 12:59:27,042 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:27,042 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:27,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into fcaeccb45c3d43109745ae652c301937(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:27,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:27,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=13, startTime=1681736366393; duration=0sec 2023-04-17 12:59:27,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:27,146 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:27,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:27,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:27,190 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:27,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:27,190 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:27,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,191 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,192 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,192 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:27,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:27,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:27,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:27,212 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:27,212 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,212 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:27,217 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:27,217 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:27,217 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,217 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:27,217 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:27,217 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:27,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:27,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:27,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:27,224 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,226 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,227 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:27,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@470a0113[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:27,229 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:27,229 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:27,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:27,230 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:27,230 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,230 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:27,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:27,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,231 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:27,231 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:27,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:27,234 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:27,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:27,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:27,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:27,246 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:27,251 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,251 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,251 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,252 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,256 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736427253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,260 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:27,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,260 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,262 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:27,262 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:27,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:27,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736427325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736427326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736427326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736427326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,362 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:27,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:27,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:27,380 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:27,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,380 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:27,381 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:27,381 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,381 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,382 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:27,384 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:27,384 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,384 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:27,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:27,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,386 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:27,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:27,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:27,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:27,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:27,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:27,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:27,403 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,404 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:27,404 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:27,406 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,406 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,406 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:27,410 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:27,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715a05c3[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:27,430 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:27,430 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,432 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:27,434 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:27,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:27,434 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:27,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:27,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,435 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:27,438 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:27,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:27,445 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:27,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:27,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:27,445 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:27,445 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:27,447 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,447 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,447 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,448 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:27,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:27,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,450 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:27,450 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:27,450 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,451 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:27,514 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/6ae0aba5440b4e0c93a4a0a17edbed5a 2023-04-17 12:59:27,563 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:27,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:27,582 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:27,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:27,582 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:27,583 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,583 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,583 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:27,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,585 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:27,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:27,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:27,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:27,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,598 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:27,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:27,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:27,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:27,634 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,634 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:27,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:27,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:27,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:27,647 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:27,648 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,650 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:27,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7293d6a4[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:27,651 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:27,651 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,652 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,652 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:27,652 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:27,652 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:27,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,658 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:27,658 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:27,658 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:27,658 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:27,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,662 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,686 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:27,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:27,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,687 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:27,687 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:27,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:27,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736427766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,787 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:27,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:27,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:27,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:27,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:27,795 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:27,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:27,795 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:27,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,796 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,796 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:27,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,798 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:27,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:27,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:27,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:27,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:27,806 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:27,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:27,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:27,826 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,826 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:27,826 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:27,826 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:27,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:27,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:27,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:27,830 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:27,832 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,838 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:27,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f7c389[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:27,840 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:27,840 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,840 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,840 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:27,841 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,841 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:27,841 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:27,841 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:27,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:27,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,855 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:27,862 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,863 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:27,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:27,899 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:27,899 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:27,899 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:27,899 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:27,899 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:27,907 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,907 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,910 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:27,930 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,943 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,951 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:27,951 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:27,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:27,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:27,952 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:27,952 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:27,953 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:27,953 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:28,053 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:28,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:28,065 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:28,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:28,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:28,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,066 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,067 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:28,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:28,068 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,068 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,068 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:28,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:28,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:28,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/bbfde9462d2246aa994dc727933fe166 2023-04-17 12:59:28,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:28,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:28,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:28,091 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:28,091 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:28,091 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:28,092 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:28,092 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:28,092 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:28,092 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:28,092 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,092 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:28,092 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:28,137 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:28,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:28,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:28,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:28,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416242dd[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:28,165 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:28,165 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,174 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:28,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:28,174 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:28,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:28,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:28,179 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:28,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,189 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:28,189 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:28,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:28,320 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:28,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:28,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:28,320 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:28,320 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:28,321 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,321 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,321 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:28,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:28,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,322 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:28,323 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:28,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:28,325 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:28,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:28,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,325 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:28,325 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,325 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:28,426 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:28,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:28,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:28,446 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:28,446 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,446 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:28,446 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:28,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,447 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,450 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:28,452 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,453 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:28,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:28,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:28,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:28,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,458 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:28,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:28,465 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:28,466 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:28,466 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:28,466 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:28,466 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:28,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:28,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:28,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:28,470 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:28,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,473 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:28,474 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:28,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:28,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b7232f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:28,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:28,475 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,478 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:28,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:28,478 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:28,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:28,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:28,479 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:28,479 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:28,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:28,480 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:28,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:28,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:28,480 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:28,480 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:28,485 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,485 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:28,485 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,486 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,488 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:28,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:28,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:28,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:28,490 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:28,491 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,491 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:28,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:28,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:28,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:28,495 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:28,495 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:28,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/e2666d709f854160bb67cccee4722514 2023-04-17 12:59:28,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/6ae0aba5440b4e0c93a4a0a17edbed5a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a 2023-04-17 12:59:28,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a, entries=200, sequenceid=168, filesize=32.1 K 2023-04-17 12:59:28,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/bbfde9462d2246aa994dc727933fe166 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166 2023-04-17 12:59:28,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 12:59:28,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/e2666d709f854160bb67cccee4722514 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514 2023-04-17 12:59:28,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 12:59:28,595 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:28,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 4c484ccfa7891407c1464a9d303bfd85 in 2027ms, sequenceid=168, compaction requested=false 2023-04-17 12:59:28,600 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:28,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:28,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:28,618 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:28,618 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:28,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:28,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,619 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,620 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:28,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:28,621 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:28,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:28,621 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:28,621 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:28,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:28,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:28,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:28,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:28,623 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:28,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:28,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:28,624 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:28,624 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:28,735 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c38cd07c4ba5492e8cefa31b1bb5321e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c38cd07c4ba5492e8cefa31b1bb5321e 2023-04-17 12:59:28,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:28,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:28,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736368962, max timestamp=1681736368989Num uniques 859; 2023-04-17 12:59:28,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:28,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736368962, max timestamp=1681736368989Num uniques 859; 2023-04-17 12:59:28,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:29,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736429128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,187 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/d92dbb6c9b31450396e78bff4007497e 2023-04-17 12:59:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736429250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,282 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/8677224f860145ae97031bc659bb1c22 2023-04-17 12:59:29,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736429345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736429359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,382 DEBUG [Thread-3254] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:29,382 DEBUG [Thread-3250] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:29,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736429362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736429362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,396 DEBUG [Thread-3248] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4237 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:29,396 DEBUG [Thread-3246] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4233 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:29,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48134 deadline: 1681736429464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736429768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,813 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/633e9fb5e2844b46a9c6c1c38d9eec55 2023-04-17 12:59:29,838 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/d92dbb6c9b31450396e78bff4007497e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e 2023-04-17 12:59:29,853 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e, entries=150, sequenceid=181, filesize=25.3 K 2023-04-17 12:59:29,870 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/8677224f860145ae97031bc659bb1c22 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22 2023-04-17 12:59:29,916 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22, entries=150, sequenceid=181, filesize=11.8 K 2023-04-17 12:59:29,922 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/633e9fb5e2844b46a9c6c1c38d9eec55 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55 2023-04-17 12:59:29,954 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55, entries=150, sequenceid=181, filesize=11.8 K 2023-04-17 12:59:29,958 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 4c484ccfa7891407c1464a9d303bfd85 in 1334ms, sequenceid=181, compaction requested=true 2023-04-17 12:59:29,958 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:29,958 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:29,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:29,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:29,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:29,958 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:29,970 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:29,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:29,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cfdef6b[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:29,973 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:29,973 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:29,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,974 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:29,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:29,978 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:29,983 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:29,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:29,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:29,986 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:29,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:29,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:29,988 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:29,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:29,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:29,988 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:29,988 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:29,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:29,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:29,990 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:29,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:29,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:30,006 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:30,035 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:30,080 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:30,081 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:30,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:30,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:30,082 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:30,082 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:30,082 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:30,082 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:30,082 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:30,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:30,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:30,182 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:30,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:30,218 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:30,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:30,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:30,218 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:30,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:30,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:30,235 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,235 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,235 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:30,238 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:30,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:30,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:30,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:30,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:30,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:30,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:30,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:30,258 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:30,264 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:30,264 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:30,265 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:30,265 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:30,265 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=125.23 KB heapSize=363.39 KB 2023-04-17 12:59:30,265 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:30,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:30,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:30,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:30,266 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:30,271 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:30,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:30,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:30,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:30,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736430331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,368 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417b0d4645f5d5544189030eb6c171d7273 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b0d4645f5d5544189030eb6c171d7273 2023-04-17 12:59:30,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:30,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736430457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736430681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:30,866 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/2e7b25701f9e415380aff2f1447b6b8b 2023-04-17 12:59:30,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736430987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,384 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/2af31e1d036240f9bab025783bd11139 2023-04-17 12:59:31,503 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d8320d8ce6a44ae1956662dfab093bb3 2023-04-17 12:59:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736431496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,531 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/2e7b25701f9e415380aff2f1447b6b8b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b 2023-04-17 12:59:31,537 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b, entries=150, sequenceid=212, filesize=25.3 K 2023-04-17 12:59:31,547 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/2af31e1d036240f9bab025783bd11139 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139 2023-04-17 12:59:31,572 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139, entries=150, sequenceid=212, filesize=11.8 K 2023-04-17 12:59:31,582 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/d8320d8ce6a44ae1956662dfab093bb3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3 2023-04-17 12:59:31,599 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3, entries=150, sequenceid=212, filesize=11.8 K 2023-04-17 12:59:31,606 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~363.34 KB/372064, currentSize=53.67 KB/54960 for 4c484ccfa7891407c1464a9d303bfd85 in 1341ms, sequenceid=212, compaction requested=true 2023-04-17 12:59:31,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:31,606 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:31,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:31,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:31,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:31,610 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:31,631 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,634 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:31,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,638 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:31,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:31,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647e9ef6[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:31,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:31,662 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:31,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:31,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:31,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,686 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:31,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:31,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:31,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:31,694 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:31,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:31,782 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:31,782 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:31,782 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:31,782 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:31,782 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:31,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,814 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:31,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,818 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,834 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:31,878 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,910 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:31,910 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:31,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:31,910 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:31,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:31,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:31,911 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:31,911 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:31,914 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:31,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:32,018 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:32,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:32,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:32,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:32,046 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:32,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:32,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:32,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:32,047 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:32,047 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:32,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:32,048 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:32,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:32,048 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:32,048 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:32,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:32,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:32,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:32,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:32,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:32,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:32,050 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:32,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:32,054 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:32,055 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:32,055 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:32,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:32,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:32,560 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a437da4ede1c498f90da97a73e586c30 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a437da4ede1c498f90da97a73e586c30 2023-04-17 12:59:32,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736372721, max timestamp=1681736372758Num uniques 859; 2023-04-17 12:59:32,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:32,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736372721, max timestamp=1681736372758Num uniques 859; 2023-04-17 12:59:32,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:32,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736372721, max timestamp=1681736372758Num uniques 859; 2023-04-17 12:59:32,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:32,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:32,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736432955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,042 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/bbcabca57205435e8e9120877916a8c5 2023-04-17 12:59:33,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736433066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,106 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/b400b48a64da4bbf87619437fe1e07dc 2023-04-17 12:59:33,262 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/09530a291d654dc4ad99fb75e0621326 2023-04-17 12:59:33,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736433286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,339 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/bbcabca57205435e8e9120877916a8c5 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5 2023-04-17 12:59:33,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48124 deadline: 1681736433394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,407 DEBUG [Thread-3254] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8251 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:33,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48106 deadline: 1681736433402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,412 DEBUG [Thread-3250] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8251 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:33,414 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5, entries=150, sequenceid=223, filesize=25.3 K 2023-04-17 12:59:33,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48158 deadline: 1681736433417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,430 DEBUG [Thread-3246] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8271 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:33,433 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/b400b48a64da4bbf87619437fe1e07dc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc 2023-04-17 12:59:33,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 136.243.18.41:48122 deadline: 1681736433429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,439 DEBUG [Thread-3248] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8283 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., hostname=jenkins-hbase17.apache.org,40215,1681736213280, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 12:59:33,459 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc, entries=150, sequenceid=223, filesize=11.8 K 2023-04-17 12:59:33,460 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/09530a291d654dc4ad99fb75e0621326 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326 2023-04-17 12:59:33,482 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326, entries=150, sequenceid=223, filesize=11.8 K 2023-04-17 12:59:33,486 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4c484ccfa7891407c1464a9d303bfd85 in 1431ms, sequenceid=223, compaction requested=true 2023-04-17 12:59:33,486 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:33,486 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:33,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:33,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:33,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:33,490 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:33,498 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,499 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:33,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:33,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37595c0e[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:33,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:33,542 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,546 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,553 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,553 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:33,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:33,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:33,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:33,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:33,566 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:33,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:33,569 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:33,569 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:33,569 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:33,569 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:33,569 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:33,566 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:33,570 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:33,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,577 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,578 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,578 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:33,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:33,586 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:33,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,587 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,587 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:33,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:33,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:33,588 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:33,588 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:33,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:33,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:33,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:33,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,688 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041711918e2683bf4ea4a4127498a87c5de9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711918e2683bf4ea4a4127498a87c5de9 2023-04-17 12:59:33,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736433679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,688 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:33,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:33,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:33,749 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:33,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:33,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:33,749 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:33,755 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/b01d5f5fe35a419295c4188c4e3162c3 2023-04-17 12:59:33,762 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,762 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,763 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:33,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,764 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,764 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,764 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:33,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:33,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:33,790 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:33,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:33,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:33,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:33,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:33,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:33,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:33,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:33,799 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:33,799 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,799 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:33,799 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:33,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:33,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736433795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,810 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,810 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:33,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:33,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eaebf91[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:33,812 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,813 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,813 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,813 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:33,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:33,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:33,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:33,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:33,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:33,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:33,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:33,830 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:33,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:33,833 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:33,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:33,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:33,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:33,833 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:33,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:33,840 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:33,841 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:33,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,854 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:33,854 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,855 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:33,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:33,855 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:33,855 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:33,855 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:33,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:33,855 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:33,858 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:33,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:33,955 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:33,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:33,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:33,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:33,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:33,978 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:33,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:33,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:33,978 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:33,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:33,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:33,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,994 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:33,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:33,994 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:33,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:34,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:34,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:34,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736434009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:34,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:34,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:34,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,022 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:34,029 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:34,029 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:34,030 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,030 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:34,030 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:34,030 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:34,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:34,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:34,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:34,038 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,038 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,039 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,040 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:34,063 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:34,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d4b9398[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:34,063 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,065 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:34,066 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:34,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:34,066 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:34,068 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,068 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:34,070 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:34,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:34,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:34,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:34,070 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:34,070 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:34,071 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,071 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,072 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,073 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,074 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,075 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,075 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,075 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:34,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:34,075 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,076 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,076 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:34,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,076 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:34,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,176 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:34,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:34,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:34,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:34,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:34,184 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:34,184 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,184 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:34,184 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:34,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,186 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,187 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,187 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,188 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:34,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:34,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:34,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:34,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:34,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,189 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:34,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:34,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:34,190 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:34,190 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:34,192 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,192 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,192 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,192 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:34,194 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:34,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd225e9[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:34,194 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,203 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,205 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:34,205 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:34,205 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:34,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,206 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:34,213 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:34,213 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,213 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:34,214 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:34,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:34,217 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:34,217 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:34,217 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:34,217 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:34,217 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:34,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,229 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,231 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,232 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,237 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:34,237 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,238 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:34,238 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:34,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/50e2d0cabca847f2bb9f5b38a7e4f025 2023-04-17 12:59:34,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/71f8ee92e72749a8bb293ef97cf34b96 2023-04-17 12:59:34,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/b01d5f5fe35a419295c4188c4e3162c3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3 2023-04-17 12:59:34,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3, entries=200, sequenceid=260, filesize=32.2 K 2023-04-17 12:59:34,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/50e2d0cabca847f2bb9f5b38a7e4f025 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025 2023-04-17 12:59:34,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025, entries=150, sequenceid=260, filesize=11.9 K 2023-04-17 12:59:34,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/71f8ee92e72749a8bb293ef97cf34b96 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96 2023-04-17 12:59:34,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96, entries=150, sequenceid=260, filesize=11.9 K 2023-04-17 12:59:34,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 4c484ccfa7891407c1464a9d303bfd85 in 719ms, sequenceid=260, compaction requested=true 2023-04-17 12:59:34,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:34,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:34,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:59:34,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:59:34,321 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:59:34,321 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:59:34,328 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72522 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:59:34,329 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:34,329 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,329 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/5d6627628295453db50889c438d489c4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=70.8 K 2023-04-17 12:59:34,337 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 169662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:59:34,337 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:34,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:34,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 12:59:34,337 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:34,337 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,338 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/fcaeccb45c3d43109745ae652c301937, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=165.7 K 2023-04-17 12:59:34,338 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:34,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:34,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:34,341 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting fcaeccb45c3d43109745ae652c301937, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:34,341 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6ae0aba5440b4e0c93a4a0a17edbed5a, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736366236 2023-04-17 12:59:34,342 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting d92dbb6c9b31450396e78bff4007497e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681736366589 2023-04-17 12:59:34,342 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 2e7b25701f9e415380aff2f1447b6b8b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681736369059 2023-04-17 12:59:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:34,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:34,337 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 5d6627628295453db50889c438d489c4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:34,343 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting bbcabca57205435e8e9120877916a8c5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681736370321 2023-04-17 12:59:34,343 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting b01d5f5fe35a419295c4188c4e3162c3, keycount=200, bloomtype=ROW, size=32.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:34,352 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e2666d709f854160bb67cccee4722514, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736366236 2023-04-17 12:59:34,353 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 633e9fb5e2844b46a9c6c1c38d9eec55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681736366589 2023-04-17 12:59:34,353 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d8320d8ce6a44ae1956662dfab093bb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681736369059 2023-04-17 12:59:34,353 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 09530a291d654dc4ad99fb75e0621326, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681736370321 2023-04-17 12:59:34,354 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 71f8ee92e72749a8bb293ef97cf34b96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:34,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:34,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:34,357 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:34,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:34,357 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:34,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,358 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,360 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,360 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,361 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:34,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:34,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:34,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:34,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,364 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:34,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:34,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:34,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:34,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:34,370 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:34,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:34,370 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,381 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#481 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:34,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:34,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:34,391 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,391 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,392 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:34,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6016a371[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:34,395 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:34,395 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,395 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:34,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:34,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:34,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,396 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:34,396 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:34,397 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,397 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,406 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#480 average throughput is 0.45 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:34,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:34,407 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:34,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:34,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:34,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:34,407 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:34,408 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,408 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,408 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,409 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:34,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:34,412 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:34,412 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:34,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,415 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:34,421 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/c485ec9525734e2dbc3c4ee2b4145043 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c485ec9525734e2dbc3c4ee2b4145043 2023-04-17 12:59:34,431 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into c485ec9525734e2dbc3c4ee2b4145043(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:34,432 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:34,432 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=10, startTime=1681736374321; duration=0sec 2023-04-17 12:59:34,432 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:34,432 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 12:59:34,434 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72522 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 12:59:34,434 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:34,434 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,434 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/d686a6c2487a4831b90f4a250497f9bd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=70.8 K 2023-04-17 12:59:34,435 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting d686a6c2487a4831b90f4a250497f9bd, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681736365150 2023-04-17 12:59:34,435 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting bbfde9462d2246aa994dc727933fe166, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681736366236 2023-04-17 12:59:34,435 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 8677224f860145ae97031bc659bb1c22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681736366589 2023-04-17 12:59:34,436 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 2af31e1d036240f9bab025783bd11139, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681736369059 2023-04-17 12:59:34,436 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b400b48a64da4bbf87619437fe1e07dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681736370321 2023-04-17 12:59:34,436 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 50e2d0cabca847f2bb9f5b38a7e4f025, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:34,454 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041740f482b5c9e04684a8981934d7db2b87 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740f482b5c9e04684a8981934d7db2b87 2023-04-17 12:59:34,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736374468, max timestamp=1681736374487Num uniques 859; 2023-04-17 12:59:34,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:34,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736374468, max timestamp=1681736374487Num uniques 859; 2023-04-17 12:59:34,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:34,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681736374468, max timestamp=1681736374487Num uniques 859; 2023-04-17 12:59:34,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase17:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 12:59:34,511 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:34,512 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:34,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:34,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:34,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:34,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:34,554 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:34,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:34,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:34,554 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/f75f24ccbebe45f5a49cf44c51034355 2023-04-17 12:59:34,555 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,555 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,556 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,557 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,557 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,557 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:34,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:34,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:34,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:34,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,578 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:34,590 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:34,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:34,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:34,591 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:34,591 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:34,592 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,593 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:34,598 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:34,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bdfdad5[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:34,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:34,622 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,624 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:34,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:34,624 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:34,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,624 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:34,625 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:34,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,626 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:34,630 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:34,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736434639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c99ac84ee4ef4262870e87f0f1db5e27 2023-04-17 12:59:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:34,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:34,666 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:34,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:34,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:34,666 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:34,666 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:34,669 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:34,670 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,671 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,673 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,673 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:34,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:34,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:34,673 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:34,673 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:34,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:34,674 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:34,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:34,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:34,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:34,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:34,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/853ec065bf2c4280900a37d101efd86b 2023-04-17 12:59:34,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/f75f24ccbebe45f5a49cf44c51034355 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355 2023-04-17 12:59:34,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355, entries=150, sequenceid=271, filesize=25.4 K 2023-04-17 12:59:34,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c99ac84ee4ef4262870e87f0f1db5e27 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27 2023-04-17 12:59:34,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 12:59:34,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/853ec065bf2c4280900a37d101efd86b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b 2023-04-17 12:59:34,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:34,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736434746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 12:59:34,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4c484ccfa7891407c1464a9d303bfd85 in 421ms, sequenceid=271, compaction requested=false 2023-04-17 12:59:34,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:34,773 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:34,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:34,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:34,777 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:34,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:34,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:34,777 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,778 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,778 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:34,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:34,779 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:34,779 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:34,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:34,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:34,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:34,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:34,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:34,794 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:34,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:34,798 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:34,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:34,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:34,798 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 12:59:34,798 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:34,799 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:34,799 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:34,799 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:34,799 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:34,800 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 12:59:34,876 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304176b47f51eb57a4588bb653441244892ae to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176b47f51eb57a4588bb653441244892ae 2023-04-17 12:59:34,929 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=308, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/7a4a9f0e92cb48ec8a10111745f2d569 2023-04-17 12:59:34,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:34,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:35,019 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/ea6308e1c0094023a51bda0444bac9b4 2023-04-17 12:59:35,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736435033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,070 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/b5b888269e5d431a85f9c956f1e41491 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b5b888269e5d431a85f9c956f1e41491 2023-04-17 12:59:35,078 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into b5b888269e5d431a85f9c956f1e41491(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:35,078 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:35,078 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=10, startTime=1681736374321; duration=0sec 2023-04-17 12:59:35,078 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:35,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:35,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736435142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:35,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736435347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,383 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/533c80b8b4a84bc985fc30772081cf82 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/533c80b8b4a84bc985fc30772081cf82 2023-04-17 12:59:35,388 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into 533c80b8b4a84bc985fc30772081cf82(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 12:59:35,388 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:35,388 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=10, startTime=1681736374321; duration=1sec 2023-04-17 12:59:35,388 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:35,489 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=308 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/7c1bce89fa0b453ba1fc789ecb58f03e 2023-04-17 12:59:35,493 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/7a4a9f0e92cb48ec8a10111745f2d569 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569 2023-04-17 12:59:35,496 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569, entries=150, sequenceid=308, filesize=25.4 K 2023-04-17 12:59:35,497 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/ea6308e1c0094023a51bda0444bac9b4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4 2023-04-17 12:59:35,502 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4, entries=150, sequenceid=308, filesize=11.9 K 2023-04-17 12:59:35,502 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/7c1bce89fa0b453ba1fc789ecb58f03e as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e 2023-04-17 12:59:35,509 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e, entries=150, sequenceid=308, filesize=11.9 K 2023-04-17 12:59:35,510 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 4c484ccfa7891407c1464a9d303bfd85 in 712ms, sequenceid=308, compaction requested=true 2023-04-17 12:59:35,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:35,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:35,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:35,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:35,510 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:35,522 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,522 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,523 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,523 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:35,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad95165[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:35,524 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:35,524 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,525 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,525 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:35,525 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:35,525 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,526 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:35,526 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:35,526 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:35,527 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:35,528 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,528 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,528 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,529 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,531 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,543 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,543 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,544 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:35,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,544 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,544 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:35,544 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:35,645 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:35,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:35,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:35,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:35,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:35,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:35,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:35,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:35,667 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,667 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,668 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:35,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:35,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:35,679 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:35,679 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:35,679 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,679 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:35,680 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:35,680 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:35,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:35,680 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,681 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:35,681 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:35,683 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,685 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,686 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,686 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2628d4ba[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:35,689 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:35,690 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,690 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:35,690 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:35,691 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:35,691 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,691 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:35,692 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,692 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:35,692 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,693 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:35,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:35,694 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:35,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:35,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:35,694 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:35,694 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:35,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,695 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,696 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,697 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,700 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,700 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,700 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:35,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:35,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:35,700 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,701 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:35,701 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:35,700 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,701 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,718 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041775d679b530554880b8aefaaa82961bee to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041775d679b530554880b8aefaaa82961bee 2023-04-17 12:59:35,726 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/139e76bf59984dfa91bab7c881d326dc 2023-04-17 12:59:35,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/e1954b421d024c2c97b5c239f38b3780 2023-04-17 12:59:35,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:35,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736435765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,806 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:35,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:35,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:35,809 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:35,809 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,809 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:35,809 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:35,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,810 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,810 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,811 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:35,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:35,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:35,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:35,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:35,812 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:35,813 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,813 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:35,813 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:35,813 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:35,813 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:35,814 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,815 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:35,818 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:35,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7370e625[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:35,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:35,822 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,823 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,823 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:35,824 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:35,824 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:35,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,824 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:35,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:35,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,826 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:35,830 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:35,833 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:35,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:35,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:35,833 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:35,833 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:35,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,834 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,835 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,836 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,842 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:35,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,842 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,843 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:35,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,843 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:35,843 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:35,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,846 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:35,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:35,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736435868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,943 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:35,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:35,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:35,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:35,959 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:35,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:35,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:35,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,959 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,960 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:35,960 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:35,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:35,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,961 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:35,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:35,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:35,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:35,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:35,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:35,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:35,974 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:35,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:35,982 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:35,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:35,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:35,983 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:35,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:35,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:35,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:35,983 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:35,984 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,985 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,985 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:35,986 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76fc83ec[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:35,987 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:35,987 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,988 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:35,988 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:35,988 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:35,988 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:35,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:35,989 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:35,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:35,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:35,998 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:36,002 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:36,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:36,004 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:36,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:36,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:36,005 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:36,005 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:36,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,006 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,007 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,008 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,009 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,011 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:36,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:36,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:36,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,011 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:36,011 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,014 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:36,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:36,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736436074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,111 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:36,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:36,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:36,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:36,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:36,126 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:36,126 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,126 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:36,126 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:36,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,130 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,131 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,131 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:36,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:36,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:36,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:36,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:36,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:36,138 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:36,138 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,138 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:36,139 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:36,139 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:36,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:36,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:36,139 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:36,140 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,140 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,141 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:36,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16cea62f[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:36,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:36,142 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,143 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,144 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,144 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:36,144 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:36,144 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:36,144 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:36,145 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,145 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:36,145 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:36,145 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:36,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:36,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:36,148 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:36,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:36,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:36,148 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:36,148 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:36,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,149 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,151 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,153 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,156 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,157 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,157 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:36,157 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:36,157 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:36,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/61109d547a7545b39d275536aacb3ff2 2023-04-17 12:59:36,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/139e76bf59984dfa91bab7c881d326dc as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc 2023-04-17 12:59:36,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc, entries=150, sequenceid=322, filesize=25.4 K 2023-04-17 12:59:36,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/e1954b421d024c2c97b5c239f38b3780 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780 2023-04-17 12:59:36,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780, entries=150, sequenceid=322, filesize=11.9 K 2023-04-17 12:59:36,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/61109d547a7545b39d275536aacb3ff2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2 2023-04-17 12:59:36,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2, entries=150, sequenceid=322, filesize=11.9 K 2023-04-17 12:59:36,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4c484ccfa7891407c1464a9d303bfd85 in 550ms, sequenceid=322, compaction requested=true 2023-04-17 12:59:36,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:36,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:36,210 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:36,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:36,210 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:36,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:36,211 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:36,211 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:36,211 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49104 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:36,211 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,211 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:36,211 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/533c80b8b4a84bc985fc30772081cf82, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=102.1 K 2023-04-17 12:59:36,211 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,212 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b5b888269e5d431a85f9c956f1e41491, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=48.0 K 2023-04-17 12:59:36,212 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting b5b888269e5d431a85f9c956f1e41491, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:36,212 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 533c80b8b4a84bc985fc30772081cf82, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:36,213 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting c99ac84ee4ef4262870e87f0f1db5e27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736373645 2023-04-17 12:59:36,213 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting f75f24ccbebe45f5a49cf44c51034355, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736373645 2023-04-17 12:59:36,213 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ea6308e1c0094023a51bda0444bac9b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736374584 2023-04-17 12:59:36,214 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e1954b421d024c2c97b5c239f38b3780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:36,215 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7a4a9f0e92cb48ec8a10111745f2d569, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736374584 2023-04-17 12:59:36,216 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 139e76bf59984dfa91bab7c881d326dc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:36,241 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#492 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:36,258 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:36,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:36,282 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#493 average throughput is 0.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:36,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:36,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:36,283 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:36,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:36,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:36,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,284 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,285 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:36,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:36,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:36,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:36,323 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:36,323 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,323 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:36,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:36,347 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:36,347 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,347 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:36,347 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:59:36,347 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:36,348 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,348 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:36,348 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,348 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:36,348 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:36,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:36,394 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417321b7c9878304c9cb73adc6ab96737e6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417321b7c9878304c9cb73adc6ab96737e6 2023-04-17 12:59:36,407 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/544bf265d8864a1e82b5ccd11ba1f4f6 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/544bf265d8864a1e82b5ccd11ba1f4f6 2023-04-17 12:59:36,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:36,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736436409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,412 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into 544bf265d8864a1e82b5ccd11ba1f4f6(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:36,412 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:36,412 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=12, startTime=1681736376210; duration=0sec 2023-04-17 12:59:36,412 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:36,412 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 12:59:36,413 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49104 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 12:59:36,413 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:36,414 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,414 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c485ec9525734e2dbc3c4ee2b4145043, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=48.0 K 2023-04-17 12:59:36,414 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c485ec9525734e2dbc3c4ee2b4145043, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1681736372919 2023-04-17 12:59:36,415 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 853ec065bf2c4280900a37d101efd86b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681736373645 2023-04-17 12:59:36,416 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=342, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/faff55aa47f94da1bacb92c3c3432c6b 2023-04-17 12:59:36,416 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 7c1bce89fa0b453ba1fc789ecb58f03e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=308, earliestPutTs=1681736374584 2023-04-17 12:59:36,416 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 61109d547a7545b39d275536aacb3ff2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:36,444 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#496 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:36,447 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c7cf9cfa04f048c1bda70a4ba0c12ea7 2023-04-17 12:59:36,482 WARN [DataStreamer for file /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/c528eb5ccf544a63962a8399043065b7] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-17 12:59:36,491 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/c528eb5ccf544a63962a8399043065b7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c528eb5ccf544a63962a8399043065b7 2023-04-17 12:59:36,505 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/866f49748255437fae6198ba027c7acd 2023-04-17 12:59:36,510 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into c528eb5ccf544a63962a8399043065b7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:36,510 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:36,510 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=12, startTime=1681736376210; duration=0sec 2023-04-17 12:59:36,510 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:36,510 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/faff55aa47f94da1bacb92c3c3432c6b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b 2023-04-17 12:59:36,513 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b, entries=150, sequenceid=342, filesize=25.4 K 2023-04-17 12:59:36,514 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c7cf9cfa04f048c1bda70a4ba0c12ea7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7 2023-04-17 12:59:36,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:36,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736436513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,526 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7, entries=150, sequenceid=342, filesize=11.9 K 2023-04-17 12:59:36,531 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/866f49748255437fae6198ba027c7acd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd 2023-04-17 12:59:36,540 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd, entries=150, sequenceid=342, filesize=11.9 K 2023-04-17 12:59:36,541 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4c484ccfa7891407c1464a9d303bfd85 in 194ms, sequenceid=342, compaction requested=false 2023-04-17 12:59:36,541 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:36,541 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:36,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:36,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:36,541 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:36,554 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,554 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,556 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,556 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:36,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a32d48[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:36,557 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:36,557 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,558 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:36,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:36,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:36,559 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,559 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:36,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:36,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:36,559 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:36,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:36,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:36,564 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:36,564 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:36,564 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:36,564 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:36,564 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:36,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:36,565 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:36,567 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,568 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,571 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,571 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:36,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,571 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:36,571 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:36,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:36,571 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,572 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:36,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,572 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:36,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:36,572 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:36,672 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:36,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:36,675 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:36,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:36,675 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:36,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,676 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:36,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:36,677 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,677 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,677 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:36,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:36,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:36,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:36,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:36,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:36,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:36,678 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:36,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:36,683 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:36,683 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:36,684 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:36,684 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:36,688 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/5404e7e7452148d6b21318472ea5a766 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/5404e7e7452148d6b21318472ea5a766 2023-04-17 12:59:36,693 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into 5404e7e7452148d6b21318472ea5a766(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:36,693 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:36,693 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=12, startTime=1681736376210; duration=0sec 2023-04-17 12:59:36,693 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:36,707 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179a56897eb93642dd9a575982f7fa9eb3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179a56897eb93642dd9a575982f7fa9eb3 2023-04-17 12:59:36,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:36,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:36,725 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/ad038b5259914b21bd1e01abc1a3fdd2 2023-04-17 12:59:36,773 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/8fbc9e8c919846e0ab21fe0bfe1184a0 2023-04-17 12:59:36,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736436799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:36,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:36,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736436901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:37,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736437105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,250 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/ec720c2aa099405db2df835624fc16dd 2023-04-17 12:59:37,254 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/ad038b5259914b21bd1e01abc1a3fdd2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2 2023-04-17 12:59:37,258 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2, entries=150, sequenceid=356, filesize=25.4 K 2023-04-17 12:59:37,258 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/8fbc9e8c919846e0ab21fe0bfe1184a0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0 2023-04-17 12:59:37,271 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0, entries=150, sequenceid=356, filesize=11.9 K 2023-04-17 12:59:37,271 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/ec720c2aa099405db2df835624fc16dd as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd 2023-04-17 12:59:37,277 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd, entries=150, sequenceid=356, filesize=11.9 K 2023-04-17 12:59:37,278 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4c484ccfa7891407c1464a9d303bfd85 in 594ms, sequenceid=356, compaction requested=true 2023-04-17 12:59:37,278 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:37,278 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:37,278 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:37,278 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:37,278 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,279 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:37,280 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:37,280 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:37,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:37,280 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:37,281 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,281 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296bc77[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:37,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:37,282 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,282 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:37,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:37,282 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:37,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,283 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:37,283 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:37,283 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:37,283 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:37,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:37,283 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:37,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:37,286 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:37,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:37,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:37,286 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:37,286 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:37,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:37,288 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,289 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,291 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:37,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:37,291 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:37,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:37,291 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:37,291 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:37,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:37,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:37,392 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:37,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:37,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:37,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:37,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:37,397 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:37,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:37,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:37,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:37,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,397 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:37,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:37,398 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:37,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:37,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:37,399 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,399 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:37,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:37,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:37,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:37,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:37,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:37,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:37,400 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:37,402 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:37,402 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:37,402 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:37,402 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:37,402 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 12:59:37,402 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:37,402 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:37,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:37,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:37,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:37,403 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:37,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:37,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:37,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736437444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,470 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171d6957da9bb04f51bf22700d380f3a1d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171d6957da9bb04f51bf22700d380f3a1d 2023-04-17 12:59:37,493 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/41e71c283f1246cd832cc3415ac8c2f3 2023-04-17 12:59:37,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:37,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736437551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:37,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736437753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:37,943 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/159da91c6e044770b96d960c02b1346d 2023-04-17 12:59:37,996 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/8b2c608b59b349da9cb808175e24def7 2023-04-17 12:59:38,001 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/41e71c283f1246cd832cc3415ac8c2f3 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3 2023-04-17 12:59:38,012 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3, entries=150, sequenceid=377, filesize=25.4 K 2023-04-17 12:59:38,013 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/159da91c6e044770b96d960c02b1346d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d 2023-04-17 12:59:38,023 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d, entries=150, sequenceid=377, filesize=11.9 K 2023-04-17 12:59:38,026 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/8b2c608b59b349da9cb808175e24def7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7 2023-04-17 12:59:38,030 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7, entries=150, sequenceid=377, filesize=11.9 K 2023-04-17 12:59:38,034 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4c484ccfa7891407c1464a9d303bfd85 in 632ms, sequenceid=377, compaction requested=true 2023-04-17 12:59:38,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:38,034 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:38,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:38,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,034 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:38,035 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:38,036 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,036 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,036 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,036 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:38,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e7ac2a[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:38,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:38,038 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,041 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,041 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:38,041 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:38,041 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:38,041 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,041 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:38,042 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:38,042 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,043 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:38,043 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:38,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:38,046 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:38,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:38,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:38,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:38,046 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:38,048 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,071 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,070 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,077 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:38,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:38,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,084 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,084 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:38,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:38,085 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:38,085 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:38,085 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,085 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:38,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:38,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40215] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736438176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,185 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:38,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:38,196 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:38,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:38,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:38,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,196 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,197 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,197 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,198 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,199 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:38,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:38,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:38,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:38,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,199 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:38,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:38,218 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:38,218 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:38,218 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:38,218 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:38,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:38,218 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,219 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:38,219 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:38,220 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,220 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,221 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,221 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:38,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737b8fc7[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:38,223 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:38,223 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,224 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,224 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:38,224 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:38,224 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:38,224 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,225 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:38,225 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:38,225 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,225 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,225 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:38,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:38,229 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:38,229 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:38,229 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:38,229 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:38,229 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:38,231 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,231 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,231 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,232 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,234 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,239 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,239 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:38,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:38,240 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:38,240 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:38,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736438281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,340 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:38,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:38,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:38,350 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:38,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:38,350 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:38,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,351 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,352 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,352 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,353 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,353 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:38,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:38,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:38,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:38,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,383 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:38,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:38,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:38,384 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,384 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:38,384 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:38,384 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:38,384 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:38,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:38,385 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:38,386 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,387 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,388 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:38,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5d258d[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:38,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:38,392 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,393 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,393 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:38,396 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:38,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:38,396 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:38,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:38,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,398 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:38,398 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:38,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:38,402 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:38,402 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:38,402 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:38,402 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:38,402 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:38,403 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,403 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,403 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,405 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,407 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,411 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:38,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,411 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,411 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:38,411 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:38,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:38,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:38,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736438487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,515 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:38,530 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041783d1519c638a4c409102c10064d30b38 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041783d1519c638a4c409102c10064d30b38 2023-04-17 12:59:38,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:38,538 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:38,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:38,538 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:38,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,539 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,540 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,540 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,540 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,540 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:38,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:38,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:38,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,542 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:38,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:38,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:38,550 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:38,550 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:38,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,552 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,552 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:38,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b1f4775[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:38,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:38,554 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,555 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,555 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,555 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:38,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:38,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:38,558 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:38,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:38,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,560 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:38,560 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:38,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:38,573 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:38,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:38,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:38,573 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:38,573 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:38,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:38,574 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,575 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,576 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,579 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:38,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:38,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,579 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:38,579 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,599 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/a95b0f9ac44b4f23921f9c081efb173b 2023-04-17 12:59:38,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/74d54f05e43a4387ba6fcf8d4276c518 2023-04-17 12:59:38,679 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:38,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:38,730 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:38,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:38,730 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:38,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,737 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,750 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,754 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:38,754 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,758 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:38,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:38,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:38,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/6f75dfe9664141158a47d70751893130 2023-04-17 12:59:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:38,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:38,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:38,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:38,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:38,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:38,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:38,785 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,785 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:38,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:38,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:38,786 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:38,794 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,795 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/a95b0f9ac44b4f23921f9c081efb173b as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b 2023-04-17 12:59:38,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:38,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736438795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,803 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,803 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b, entries=200, sequenceid=390, filesize=32.3 K 2023-04-17 12:59:38,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:38,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b428c0[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:38,826 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:38,826 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/74d54f05e43a4387ba6fcf8d4276c518 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518 2023-04-17 12:59:38,838 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:38,839 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:38,839 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:38,839 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:38,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,850 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:38,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518, entries=150, sequenceid=390, filesize=11.9 K 2023-04-17 12:59:38,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/6f75dfe9664141158a47d70751893130 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130 2023-04-17 12:59:38,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:38,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:38,862 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:38,866 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:38,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130, entries=150, sequenceid=390, filesize=11.9 K 2023-04-17 12:59:38,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4c484ccfa7891407c1464a9d303bfd85 in 814ms, sequenceid=390, compaction requested=true 2023-04-17 12:59:38,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:38,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:38,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 12:59:38,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 12:59:38,895 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:59:38,896 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:59:38,896 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:38,896 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,896 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/544bf265d8864a1e82b5ccd11ba1f4f6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=134.6 K 2023-04-17 12:59:38,897 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 544bf265d8864a1e82b5ccd11ba1f4f6, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:38,898 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:59:38,902 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting faff55aa47f94da1bacb92c3c3432c6b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681736375755 2023-04-17 12:59:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,906 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting ad038b5259914b21bd1e01abc1a3fdd2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681736376399 2023-04-17 12:59:38,907 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 41e71c283f1246cd832cc3415ac8c2f3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681736376757 2023-04-17 12:59:38,907 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting a95b0f9ac44b4f23921f9c081efb173b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377432 2023-04-17 12:59:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,926 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:59:38,927 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:38,927 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:38,927 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c528eb5ccf544a63962a8399043065b7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=60.0 K 2023-04-17 12:59:38,934 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c528eb5ccf544a63962a8399043065b7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:38,938 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 866f49748255437fae6198ba027c7acd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681736375755 2023-04-17 12:59:38,946 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting ec720c2aa099405db2df835624fc16dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681736376399 2023-04-17 12:59:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:38,954 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8b2c608b59b349da9cb808175e24def7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681736376757 2023-04-17 12:59:38,960 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 6f75dfe9664141158a47d70751893130, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377438 2023-04-17 12:59:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:38,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:38,982 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:38,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:38,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:38,982 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:38,983 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:39,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,010 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:39,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:39,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:39,023 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:39,037 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#507 average throughput is 0.30 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:39,046 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:39,090 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#508 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:39,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:39,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:39,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:39,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:39,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:39,102 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:39,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:39,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:39,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:39,103 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,106 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:39,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:39,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:39,108 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:39,108 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:39,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:39,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:39,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 12:59:39,177 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/2f80677df7de480ebc2de187d3d4affe as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/2f80677df7de480ebc2de187d3d4affe 2023-04-17 12:59:39,203 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into 2f80677df7de480ebc2de187d3d4affe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:39,203 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:39,203 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=11, startTime=1681736378895; duration=0sec 2023-04-17 12:59:39,203 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:39,203 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 12:59:39,207 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 12:59:39,207 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:39,207 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:39,207 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/5404e7e7452148d6b21318472ea5a766, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=60.0 K 2023-04-17 12:59:39,208 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:39,210 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 5404e7e7452148d6b21318472ea5a766, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681736375023 2023-04-17 12:59:39,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:39,217 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting c7cf9cfa04f048c1bda70a4ba0c12ea7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681736375755 2023-04-17 12:59:39,218 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 8fbc9e8c919846e0ab21fe0bfe1184a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681736376399 2023-04-17 12:59:39,218 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 159da91c6e044770b96d960c02b1346d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681736376757 2023-04-17 12:59:39,219 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 74d54f05e43a4387ba6fcf8d4276c518, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377438 2023-04-17 12:59:39,219 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/3ef3285b57984ec98ae5298eb76b5035 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3ef3285b57984ec98ae5298eb76b5035 2023-04-17 12:59:39,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(933): Client=jenkins//136.243.18.41 procedure request for: flush-table-proc 2023-04-17 12:59:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 12:59:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 12:59:39,238 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 12:59:39,238 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:39,238 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 12:59:39,238 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 12:59:39,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,250 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:39,258 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,259 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:39,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:39,262 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,263 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 12:59:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:39,266 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into 3ef3285b57984ec98ae5298eb76b5035(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:39,266 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:39,266 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=11, startTime=1681736378894; duration=0sec 2023-04-17 12:59:39,267 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:39,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:39,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 12:59:39,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:39,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 12:59:39,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 12:59:39,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 12:59:39,276 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 12:59:39,276 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 12:59:39,276 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 12:59:39,277 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 12:59:39,278 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 12:59:39,278 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:39,278 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. started... 2023-04-17 12:59:39,292 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#509 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:39,294 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 12:59:39,294 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:39,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:39,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:39,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:39,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:39,295 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:39,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. as already flushing 2023-04-17 12:59:39,371 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/0a001301baac41f591297c4d5d68c424 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/0a001301baac41f591297c4d5d68c424 2023-04-17 12:59:39,391 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into 0a001301baac41f591297c4d5d68c424(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:39,391 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:39,391 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=11, startTime=1681736378895; duration=0sec 2023-04-17 12:59:39,392 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:39,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736439387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,402 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f7c2d7f4c9044bfbb4c3da42ec109a13 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f7c2d7f4c9044bfbb4c3da42ec109a13 2023-04-17 12:59:39,456 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=411, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/229b7918d63b4bc0a77070ee5e55409f 2023-04-17 12:59:39,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40215] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736439497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736439711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:39,970 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/e9f7480039c4480e977aed24ae7a0aa2 2023-04-17 12:59:40,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:40,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736440026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,455 DEBUG [Thread-3265] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fd90ac8 to 127.0.0.1:62647 2023-04-17 12:59:40,456 DEBUG [Thread-3265] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:40,456 DEBUG [Thread-3257] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b8bf14 to 127.0.0.1:62647 2023-04-17 12:59:40,456 DEBUG [Thread-3257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:40,458 DEBUG [Thread-3261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c00a18c to 127.0.0.1:62647 2023-04-17 12:59:40,458 DEBUG [Thread-3261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:40,459 DEBUG [Thread-3259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x387217bb to 127.0.0.1:62647 2023-04-17 12:59:40,459 DEBUG [Thread-3259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:40,459 DEBUG [Thread-3263] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3da68fe0 to 127.0.0.1:62647 2023-04-17 12:59:40,459 DEBUG [Thread-3263] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:40,523 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/bb79007e746a4f7e8d36d8bd039e0c97 2023-04-17 12:59:40,527 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/229b7918d63b4bc0a77070ee5e55409f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f 2023-04-17 12:59:40,531 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f, entries=150, sequenceid=411, filesize=25.4 K 2023-04-17 12:59:40,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 12:59:40,532 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/e9f7480039c4480e977aed24ae7a0aa2 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2 2023-04-17 12:59:40,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 136.243.18.41:48134 deadline: 1681736440531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,536 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2, entries=150, sequenceid=411, filesize=11.9 K 2023-04-17 12:59:40,537 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/bb79007e746a4f7e8d36d8bd039e0c97 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97 2023-04-17 12:59:40,541 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97, entries=150, sequenceid=411, filesize=11.9 K 2023-04-17 12:59:40,542 INFO [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4c484ccfa7891407c1464a9d303bfd85 in 1248ms, sequenceid=411, compaction requested=false 2023-04-17 12:59:40,543 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:40,543 DEBUG [rs(jenkins-hbase17.apache.org,40215,1681736213280)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:40,543 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 12:59:40,543 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 12:59:40,543 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,543 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 12:59:40,543 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 12:59:40,667 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,667 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:40,668 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,668 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 12:59:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:40,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase17.apache.org,40215,1681736213280' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 12:59:40,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291e8a66[Count = 0] remaining members to acquire global barrier 2023-04-17 12:59:40,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 12:59:40,669 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,670 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 12:59:40,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 12:59:40,670 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase17.apache.org,40215,1681736213280' in zk 2023-04-17 12:59:40,671 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,671 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 12:59:40,671 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 12:59:40,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:40,671 DEBUG [member: 'jenkins-hbase17.apache.org,40215,1681736213280' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 12:59:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase17.apache.org,40215,1681736213280': 2023-04-17 12:59:40,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase17.apache.org,40215,1681736213280' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 12:59:40,674 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 12:59:40,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 12:59:40,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 12:59:40,674 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 12:59:40,674 INFO [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 12:59:40,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 12:59:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 12:59:40,675 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 12:59:40,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,676 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 12:59:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 12:59:40,677 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 12:59:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 12:59:40,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 12:59:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 12:59:40,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 12:59:40,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 12:59:40,679 DEBUG [(jenkins-hbase17.apache.org,44317,1681736211069)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,679 DEBUG [Thread-3256] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 12:59:40,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 12:59:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 12:59:40,679 DEBUG [Thread-3256] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 12:59:40,679 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 12:59:40,780 DEBUG [Thread-3256] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 12:59:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 12:59:41,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40215] regionserver.HRegion(9158): Flush requested on 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:41,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:41,537 DEBUG [Thread-3252] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a1010a6 to 127.0.0.1:62647 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:41,537 DEBUG [Thread-3252] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:41,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:41,950 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f1e5444495634f5a94a19307c44c4b85 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f1e5444495634f5a94a19307c44c4b85 2023-04-17 12:59:41,960 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/3542d603933b48f2803821884951fbf4 2023-04-17 12:59:41,977 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/403ecc641b1c47efac8c901280ae2a21 2023-04-17 12:59:42,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/3ba6608d636b4a91ae413bdcb83e27c0 2023-04-17 12:59:42,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/3542d603933b48f2803821884951fbf4 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4 2023-04-17 12:59:42,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4, entries=150, sequenceid=426, filesize=25.4 K 2023-04-17 12:59:42,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/403ecc641b1c47efac8c901280ae2a21 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21 2023-04-17 12:59:42,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21, entries=150, sequenceid=426, filesize=11.9 K 2023-04-17 12:59:42,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/3ba6608d636b4a91ae413bdcb83e27c0 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0 2023-04-17 12:59:42,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0, entries=150, sequenceid=426, filesize=11.9 K 2023-04-17 12:59:42,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 4c484ccfa7891407c1464a9d303bfd85 in 492ms, sequenceid=426, compaction requested=true 2023-04-17 12:59:42,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:42,029 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:42,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:42,030 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:42,031 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/A is initiating minor compaction (all files) 2023-04-17 12:59:42,031 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/A in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:42,031 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3ef3285b57984ec98ae5298eb76b5035, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=77.0 K 2023-04-17 12:59:42,031 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3ef3285b57984ec98ae5298eb76b5035, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377438 2023-04-17 12:59:42,032 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 229b7918d63b4bc0a77070ee5e55409f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681736378151 2023-04-17 12:59:42,032 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] compactions.Compactor(207): Compacting 3542d603933b48f2803821884951fbf4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681736379349 2023-04-17 12:59:42,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:42,032 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:42,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:42,035 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:42,035 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/B is initiating minor compaction (all files) 2023-04-17 12:59:42,035 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/B in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:42,035 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/0a001301baac41f591297c4d5d68c424, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=36.4 K 2023-04-17 12:59:42,036 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 0a001301baac41f591297c4d5d68c424, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377438 2023-04-17 12:59:42,037 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting e9f7480039c4480e977aed24ae7a0aa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681736378151 2023-04-17 12:59:42,038 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 403ecc641b1c47efac8c901280ae2a21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681736379349 2023-04-17 12:59:42,060 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#B#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:42,067 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#A#compaction#516 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:42,077 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/21ff098f81504610baf8de164be98379 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/21ff098f81504610baf8de164be98379 2023-04-17 12:59:42,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/B of 4c484ccfa7891407c1464a9d303bfd85 into 21ff098f81504610baf8de164be98379(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:42,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:42,082 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/B, priority=13, startTime=1681736382031; duration=0sec 2023-04-17 12:59:42,082 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 12:59:42,083 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 12:59:42,084 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 12:59:42,084 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1912): 4c484ccfa7891407c1464a9d303bfd85/C is initiating minor compaction (all files) 2023-04-17 12:59:42,084 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c484ccfa7891407c1464a9d303bfd85/C in TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:42,084 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/2f80677df7de480ebc2de187d3d4affe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0] into tmpdir=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp, totalSize=36.4 K 2023-04-17 12:59:42,085 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 2f80677df7de480ebc2de187d3d4affe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681736377438 2023-04-17 12:59:42,086 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting bb79007e746a4f7e8d36d8bd039e0c97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1681736378151 2023-04-17 12:59:42,086 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] compactions.Compactor(207): Compacting 3ba6608d636b4a91ae413bdcb83e27c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681736379349 2023-04-17 12:59:42,107 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c484ccfa7891407c1464a9d303bfd85#C#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 12:59:42,129 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/9520db1f917743228bbc796db62b135a as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/9520db1f917743228bbc796db62b135a 2023-04-17 12:59:42,130 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/1578b9e6b26b4b5287fdbe8d13f29d3d as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1578b9e6b26b4b5287fdbe8d13f29d3d 2023-04-17 12:59:42,142 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/A of 4c484ccfa7891407c1464a9d303bfd85 into 1578b9e6b26b4b5287fdbe8d13f29d3d(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:42,142 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:42,142 INFO [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/A, priority=13, startTime=1681736382029; duration=0sec 2023-04-17 12:59:42,143 DEBUG [RS:0;jenkins-hbase17:40215-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:42,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c484ccfa7891407c1464a9d303bfd85/C of 4c484ccfa7891407c1464a9d303bfd85 into 9520db1f917743228bbc796db62b135a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 12:59:42,147 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:42,147 INFO [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85., storeName=4c484ccfa7891407c1464a9d303bfd85/C, priority=13, startTime=1681736382033; duration=0sec 2023-04-17 12:59:42,148 DEBUG [RS:0;jenkins-hbase17:40215-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 12:59:43,435 DEBUG [Thread-3254] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4143b9ae to 127.0.0.1:62647 2023-04-17 12:59:43,435 DEBUG [Thread-3254] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:43,452 DEBUG [Thread-3250] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f11e84a to 127.0.0.1:62647 2023-04-17 12:59:43,452 DEBUG [Thread-3250] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:43,471 DEBUG [Thread-3248] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5be55982 to 127.0.0.1:62647 2023-04-17 12:59:43,471 DEBUG [Thread-3248] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:43,508 DEBUG [Thread-3246] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5c51ec to 127.0.0.1:62647 2023-04-17 12:59:43,508 DEBUG [Thread-3246] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:43,508 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 12:59:43,508 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 253 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2254 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2350 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2265 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2305 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(398): read 2235 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 12:59:43,509 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:59:43,509 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c217430 to 127.0.0.1:62647 2023-04-17 12:59:43,509 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:43,510 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 12:59:43,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$11(2418): Client=jenkins//136.243.18.41 disable TestAcidGuarantees 2023-04-17 12:59:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:43,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:43,513 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736383513"}]},"ts":"1681736383513"} 2023-04-17 12:59:43,514 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 12:59:43,577 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 12:59:43,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, UNASSIGN}] 2023-04-17 12:59:43,580 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, UNASSIGN 2023-04-17 12:59:43,581 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=CLOSING, regionLocation=jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:43,581 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736383581"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681736383581"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681736383581"}]},"ts":"1681736383581"} 2023-04-17 12:59:43,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280}] 2023-04-17 12:59:43,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:43,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(109): Close 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:43,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 4c484ccfa7891407c1464a9d303bfd85, disabling compactions & flushes 2023-04-17 12:59:43,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. after waiting 0 ms 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:43,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 4c484ccfa7891407c1464a9d303bfd85 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=A 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=B 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c484ccfa7891407c1464a9d303bfd85, store=C 2023-04-17 12:59:43,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 12:59:43,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:44,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:44,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041718bb9079b7d745289c21e4e2d0d973f0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041718bb9079b7d745289c21e4e2d0d973f0 2023-04-17 12:59:44,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/2dc538f4876f42adabc89a4d07d91535 2023-04-17 12:59:44,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:44,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c7e7c6c4d1b34486b0bd9352717bf154 2023-04-17 12:59:45,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/aab4a72ece644e3bb021b9780205ff55 2023-04-17 12:59:45,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/A/2dc538f4876f42adabc89a4d07d91535 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2dc538f4876f42adabc89a4d07d91535 2023-04-17 12:59:45,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2dc538f4876f42adabc89a4d07d91535, entries=150, sequenceid=436, filesize=25.4 K 2023-04-17 12:59:45,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/B/c7e7c6c4d1b34486b0bd9352717bf154 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7e7c6c4d1b34486b0bd9352717bf154 2023-04-17 12:59:45,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7e7c6c4d1b34486b0bd9352717bf154, entries=150, sequenceid=436, filesize=11.9 K 2023-04-17 12:59:45,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/.tmp/C/aab4a72ece644e3bb021b9780205ff55 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/aab4a72ece644e3bb021b9780205ff55 2023-04-17 12:59:45,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/aab4a72ece644e3bb021b9780205ff55, entries=150, sequenceid=436, filesize=11.9 K 2023-04-17 12:59:45,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 4c484ccfa7891407c1464a9d303bfd85 in 1294ms, sequenceid=436, compaction requested=false 2023-04-17 12:59:45,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ca1cf3a871834c2a9944a9060b609d87, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/fcaeccb45c3d43109745ae652c301937, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/533c80b8b4a84bc985fc30772081cf82, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/544bf265d8864a1e82b5ccd11ba1f4f6, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3ef3285b57984ec98ae5298eb76b5035, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4] to archive 2023-04-17 12:59:45,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:45,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/e239a727021c407a9acabe0c7cb3e466 2023-04-17 12:59:45,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1e4f73e6cf6041a3840cddab7223902e 2023-04-17 12:59:45,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ca1cf3a871834c2a9944a9060b609d87 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ca1cf3a871834c2a9944a9060b609d87 2023-04-17 12:59:45,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/46a5956262374967bdc2d81bd687f0f7 2023-04-17 12:59:45,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a73f0152e4884e85bdfcdacbfe1fb763 2023-04-17 12:59:45,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/fcaeccb45c3d43109745ae652c301937 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/fcaeccb45c3d43109745ae652c301937 2023-04-17 12:59:45,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/c5289365994e49919b8b6da96d7d5725 2023-04-17 12:59:45,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/6ae0aba5440b4e0c93a4a0a17edbed5a 2023-04-17 12:59:45,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/d92dbb6c9b31450396e78bff4007497e 2023-04-17 12:59:45,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2e7b25701f9e415380aff2f1447b6b8b 2023-04-17 12:59:45,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/bbcabca57205435e8e9120877916a8c5 2023-04-17 12:59:45,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/b01d5f5fe35a419295c4188c4e3162c3 2023-04-17 12:59:45,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/533c80b8b4a84bc985fc30772081cf82 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/533c80b8b4a84bc985fc30772081cf82 2023-04-17 12:59:45,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/f75f24ccbebe45f5a49cf44c51034355 2023-04-17 12:59:45,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/7a4a9f0e92cb48ec8a10111745f2d569 2023-04-17 12:59:45,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/544bf265d8864a1e82b5ccd11ba1f4f6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/544bf265d8864a1e82b5ccd11ba1f4f6 2023-04-17 12:59:45,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/139e76bf59984dfa91bab7c881d326dc 2023-04-17 12:59:45,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/faff55aa47f94da1bacb92c3c3432c6b 2023-04-17 12:59:45,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/ad038b5259914b21bd1e01abc1a3fdd2 2023-04-17 12:59:45,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/41e71c283f1246cd832cc3415ac8c2f3 2023-04-17 12:59:45,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/a95b0f9ac44b4f23921f9c081efb173b 2023-04-17 12:59:45,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3ef3285b57984ec98ae5298eb76b5035 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3ef3285b57984ec98ae5298eb76b5035 2023-04-17 12:59:45,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/229b7918d63b4bc0a77070ee5e55409f 2023-04-17 12:59:45,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/3542d603933b48f2803821884951fbf4 2023-04-17 12:59:45,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/6a3f29180d81417b9e67bb63bc7b5d80, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/d686a6c2487a4831b90f4a250497f9bd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b5b888269e5d431a85f9c956f1e41491, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/5404e7e7452148d6b21318472ea5a766, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/0a001301baac41f591297c4d5d68c424, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21] to archive 2023-04-17 12:59:45,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:45,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/35ee474a8b704e68a90dbcae3c804c12 2023-04-17 12:59:45,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/9e983480b9dd4a0d9a7a497097e3d6a4 2023-04-17 12:59:45,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/6a3f29180d81417b9e67bb63bc7b5d80 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/6a3f29180d81417b9e67bb63bc7b5d80 2023-04-17 12:59:45,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/639c1319876a481eb576ee2b7c7a3419 2023-04-17 12:59:45,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/7aff95a8ff594e30921e6f838da17050 2023-04-17 12:59:45,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/d686a6c2487a4831b90f4a250497f9bd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/d686a6c2487a4831b90f4a250497f9bd 2023-04-17 12:59:45,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b33d3dfb83ad44c9b7966f5f73256142 2023-04-17 12:59:45,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/bbfde9462d2246aa994dc727933fe166 2023-04-17 12:59:45,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8677224f860145ae97031bc659bb1c22 2023-04-17 12:59:45,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/2af31e1d036240f9bab025783bd11139 2023-04-17 12:59:45,115 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b400b48a64da4bbf87619437fe1e07dc 2023-04-17 12:59:45,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b5b888269e5d431a85f9c956f1e41491 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/b5b888269e5d431a85f9c956f1e41491 2023-04-17 12:59:45,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/50e2d0cabca847f2bb9f5b38a7e4f025 2023-04-17 12:59:45,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c99ac84ee4ef4262870e87f0f1db5e27 2023-04-17 12:59:45,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/ea6308e1c0094023a51bda0444bac9b4 2023-04-17 12:59:45,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/5404e7e7452148d6b21318472ea5a766 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/5404e7e7452148d6b21318472ea5a766 2023-04-17 12:59:45,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e1954b421d024c2c97b5c239f38b3780 2023-04-17 12:59:45,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7cf9cfa04f048c1bda70a4ba0c12ea7 2023-04-17 12:59:45,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/8fbc9e8c919846e0ab21fe0bfe1184a0 2023-04-17 12:59:45,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/159da91c6e044770b96d960c02b1346d 2023-04-17 12:59:45,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/0a001301baac41f591297c4d5d68c424 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/0a001301baac41f591297c4d5d68c424 2023-04-17 12:59:45,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/74d54f05e43a4387ba6fcf8d4276c518 2023-04-17 12:59:45,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/e9f7480039c4480e977aed24ae7a0aa2 2023-04-17 12:59:45,130 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/403ecc641b1c47efac8c901280ae2a21 2023-04-17 12:59:45,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/b17b63262a6844129a03ee7544fc38bd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/5d6627628295453db50889c438d489c4, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c485ec9525734e2dbc3c4ee2b4145043, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c528eb5ccf544a63962a8399043065b7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/2f80677df7de480ebc2de187d3d4affe, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0] to archive 2023-04-17 12:59:45,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 12:59:45,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5df98691e4e40518b10ec06acda04a0 2023-04-17 12:59:45,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d5c8d32aaba64b74a35940264253039b 2023-04-17 12:59:45,137 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/b17b63262a6844129a03ee7544fc38bd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/b17b63262a6844129a03ee7544fc38bd 2023-04-17 12:59:45,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3d9d6e48a6444e31a171ee47ee08e565 2023-04-17 12:59:45,140 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/dc71f7c161a94a6e8cf2a3d8c03f617b 2023-04-17 12:59:45,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/5d6627628295453db50889c438d489c4 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/5d6627628295453db50889c438d489c4 2023-04-17 12:59:45,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/1703fcc53af14d10ba5132989194968a 2023-04-17 12:59:45,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/e2666d709f854160bb67cccee4722514 2023-04-17 12:59:45,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/633e9fb5e2844b46a9c6c1c38d9eec55 2023-04-17 12:59:45,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/d8320d8ce6a44ae1956662dfab093bb3 2023-04-17 12:59:45,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/09530a291d654dc4ad99fb75e0621326 2023-04-17 12:59:45,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c485ec9525734e2dbc3c4ee2b4145043 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c485ec9525734e2dbc3c4ee2b4145043 2023-04-17 12:59:45,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/71f8ee92e72749a8bb293ef97cf34b96 2023-04-17 12:59:45,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/853ec065bf2c4280900a37d101efd86b 2023-04-17 12:59:45,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/7c1bce89fa0b453ba1fc789ecb58f03e 2023-04-17 12:59:45,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c528eb5ccf544a63962a8399043065b7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/c528eb5ccf544a63962a8399043065b7 2023-04-17 12:59:45,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/61109d547a7545b39d275536aacb3ff2 2023-04-17 12:59:45,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/866f49748255437fae6198ba027c7acd 2023-04-17 12:59:45,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/ec720c2aa099405db2df835624fc16dd 2023-04-17 12:59:45,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/8b2c608b59b349da9cb808175e24def7 2023-04-17 12:59:45,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/2f80677df7de480ebc2de187d3d4affe to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/2f80677df7de480ebc2de187d3d4affe 2023-04-17 12:59:45,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/6f75dfe9664141158a47d70751893130 2023-04-17 12:59:45,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/bb79007e746a4f7e8d36d8bd039e0c97 2023-04-17 12:59:45,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/3ba6608d636b4a91ae413bdcb83e27c0 2023-04-17 12:59:45,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits/439.seqid, newMaxSeqId=439, maxSeqId=4 2023-04-17 12:59:45,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85. 2023-04-17 12:59:45,171 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 4c484ccfa7891407c1464a9d303bfd85: 2023-04-17 12:59:45,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.UnassignRegionHandler(147): Closed 4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:45,173 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4c484ccfa7891407c1464a9d303bfd85, regionState=CLOSED 2023-04-17 12:59:45,173 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681736385173"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681736385173"}]},"ts":"1681736385173"} 2023-04-17 12:59:45,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-17 12:59:45,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 4c484ccfa7891407c1464a9d303bfd85, server=jenkins-hbase17.apache.org,40215,1681736213280 in 1.5920 sec 2023-04-17 12:59:45,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-17 12:59:45,183 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c484ccfa7891407c1464a9d303bfd85, UNASSIGN in 1.6000 sec 2023-04-17 12:59:45,184 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681736385183"}]},"ts":"1681736385183"} 2023-04-17 12:59:45,184 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 12:59:45,185 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 12:59:45,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6760 sec 2023-04-17 12:59:45,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 12:59:45,618 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-17 12:59:45,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.HMaster$5(2228): Client=jenkins//136.243.18.41 delete TestAcidGuarantees 2023-04-17 12:59:45,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,622 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 12:59:45,623 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,635 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:45,645 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C, FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits] 2023-04-17 12:59:45,650 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1578b9e6b26b4b5287fdbe8d13f29d3d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/1578b9e6b26b4b5287fdbe8d13f29d3d 2023-04-17 12:59:45,652 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2dc538f4876f42adabc89a4d07d91535 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/A/2dc538f4876f42adabc89a4d07d91535 2023-04-17 12:59:45,663 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/21ff098f81504610baf8de164be98379 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/21ff098f81504610baf8de164be98379 2023-04-17 12:59:45,666 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7e7c6c4d1b34486b0bd9352717bf154 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/B/c7e7c6c4d1b34486b0bd9352717bf154 2023-04-17 12:59:45,668 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/9520db1f917743228bbc796db62b135a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/9520db1f917743228bbc796db62b135a 2023-04-17 12:59:45,669 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/aab4a72ece644e3bb021b9780205ff55 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/C/aab4a72ece644e3bb021b9780205ff55 2023-04-17 12:59:45,672 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits/439.seqid to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85/recovered.edits/439.seqid 2023-04-17 12:59:45,672 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/.tmp/data/default/TestAcidGuarantees/4c484ccfa7891407c1464a9d303bfd85 2023-04-17 12:59:45,673 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 12:59:45,673 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:59:45,674 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 12:59:45,677 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711918e2683bf4ea4a4127498a87c5de9 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041711918e2683bf4ea4a4127498a87c5de9 2023-04-17 12:59:45,678 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041718bb9079b7d745289c21e4e2d0d973f0 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041718bb9079b7d745289c21e4e2d0d973f0 2023-04-17 12:59:45,679 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171d6957da9bb04f51bf22700d380f3a1d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171d6957da9bb04f51bf22700d380f3a1d 2023-04-17 12:59:45,680 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417273d5cea919e4714a19fb32346c3ad1d to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417273d5cea919e4714a19fb32346c3ad1d 2023-04-17 12:59:45,682 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417321b7c9878304c9cb73adc6ab96737e6 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417321b7c9878304c9cb73adc6ab96737e6 2023-04-17 12:59:45,683 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740f482b5c9e04684a8981934d7db2b87 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740f482b5c9e04684a8981934d7db2b87 2023-04-17 12:59:45,684 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176b47f51eb57a4588bb653441244892ae to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176b47f51eb57a4588bb653441244892ae 2023-04-17 12:59:45,685 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041775d679b530554880b8aefaaa82961bee to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041775d679b530554880b8aefaaa82961bee 2023-04-17 12:59:45,687 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041783d1519c638a4c409102c10064d30b38 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041783d1519c638a4c409102c10064d30b38 2023-04-17 12:59:45,688 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178daa27fd2bb444daaead53de69a11201 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178daa27fd2bb444daaead53de69a11201 2023-04-17 12:59:45,689 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417937015f921124f81a4158ef76155d2b8 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417937015f921124f81a4158ef76155d2b8 2023-04-17 12:59:45,690 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179a56897eb93642dd9a575982f7fa9eb3 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179a56897eb93642dd9a575982f7fa9eb3 2023-04-17 12:59:45,692 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a437da4ede1c498f90da97a73e586c30 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a437da4ede1c498f90da97a73e586c30 2023-04-17 12:59:45,693 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b0d4645f5d5544189030eb6c171d7273 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b0d4645f5d5544189030eb6c171d7273 2023-04-17 12:59:45,694 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c38cd07c4ba5492e8cefa31b1bb5321e to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c38cd07c4ba5492e8cefa31b1bb5321e 2023-04-17 12:59:45,695 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e6f45abeac2c4a88b3a5d9b1d2395c30 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e6f45abeac2c4a88b3a5d9b1d2395c30 2023-04-17 12:59:45,696 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f1e5444495634f5a94a19307c44c4b85 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f1e5444495634f5a94a19307c44c4b85 2023-04-17 12:59:45,698 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f6b21f64827c49a2a9c510217304070f to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f6b21f64827c49a2a9c510217304070f 2023-04-17 12:59:45,699 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f7c2d7f4c9044bfbb4c3da42ec109a13 to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f7c2d7f4c9044bfbb4c3da42ec109a13 2023-04-17 12:59:45,702 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8498692c184455a8c8cdcae66e6e53a to hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f8498692c184455a8c8cdcae66e6e53a 2023-04-17 12:59:45,702 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 12:59:45,709 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,711 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 12:59:45,713 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 12:59:45,714 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,714 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 12:59:45,714 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681736385714"}]},"ts":"9223372036854775807"} 2023-04-17 12:59:45,717 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 12:59:45,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4c484ccfa7891407c1464a9d303bfd85, NAME => 'TestAcidGuarantees,,1681736358427.4c484ccfa7891407c1464a9d303bfd85.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 12:59:45,717 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 12:59:45,717 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681736385717"}]},"ts":"9223372036854775807"} 2023-04-17 12:59:45,719 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 12:59:45,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 12:59:45,728 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 12:59:45,729 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 109 msec 2023-04-17 12:59:45,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44317] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 12:59:45,925 INFO [Listener at localhost.localdomain/43083] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-17 12:59:45,939 INFO [Listener at localhost.localdomain/43083] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 229), OpenFileDescriptor=566 (was 595), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1127 (was 1113) - SystemLoadAverage LEAK? -, ProcessCount=196 (was 195) - ProcessCount LEAK? -, AvailableMemoryMB=254 (was 254) 2023-04-17 12:59:45,939 INFO [Listener at localhost.localdomain/43083] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-17 12:59:45,939 INFO [Listener at localhost.localdomain/43083] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 12:59:45,940 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e23d53f to 127.0.0.1:62647 2023-04-17 12:59:45,940 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:45,940 DEBUG [Listener at localhost.localdomain/43083] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-17 12:59:45,941 DEBUG [Listener at localhost.localdomain/43083] util.JVMClusterUtil(257): Found active master hash=251057750, stopped=false 2023-04-17 12:59:45,941 INFO [Listener at localhost.localdomain/43083] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:59:45,941 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 12:59:45,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:59:45,942 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 12:59:45,942 INFO [Listener at localhost.localdomain/43083] procedure2.ProcedureExecutor(629): Stopping 2023-04-17 12:59:45,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 12:59:45,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 12:59:45,943 DEBUG [Listener at localhost.localdomain/43083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a61a90a to 127.0.0.1:62647 2023-04-17 12:59:45,943 DEBUG [Listener at localhost.localdomain/43083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:45,943 INFO [Listener at localhost.localdomain/43083] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase17.apache.org,40215,1681736213280' ***** 2023-04-17 12:59:45,943 INFO [Listener at localhost.localdomain/43083] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-17 12:59:45,943 INFO [RS:0;jenkins-hbase17:40215] regionserver.HeapMemoryManager(220): Stopping 2023-04-17 12:59:45,943 INFO [RS:0;jenkins-hbase17:40215] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-17 12:59:45,943 INFO [RS:0;jenkins-hbase17:40215] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-17 12:59:45,944 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(3304): Received CLOSE for 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:59:45,944 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1145): stopping server jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:45,944 DEBUG [RS:0;jenkins-hbase17:40215] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65686db2 to 127.0.0.1:62647 2023-04-17 12:59:45,944 DEBUG [RS:0;jenkins-hbase17:40215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:45,944 INFO [RS:0;jenkins-hbase17:40215] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-17 12:59:45,944 INFO [RS:0;jenkins-hbase17:40215] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-17 12:59:45,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 7a30f634bdc6e3e5b973c8313abab69a, disabling compactions & flushes 2023-04-17 12:59:45,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:59:45,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:59:45,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. after waiting 0 ms 2023-04-17 12:59:45,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:59:45,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 7a30f634bdc6e3e5b973c8313abab69a 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-17 12:59:45,945 INFO [RS:0;jenkins-hbase17:40215] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-17 12:59:45,945 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-17 12:59:45,945 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-17 12:59:45,945 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-17 12:59:45,946 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 7a30f634bdc6e3e5b973c8313abab69a=hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a.} 2023-04-17 12:59:45,947 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740, 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:59:45,948 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 12:59:45,948 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 12:59:45,949 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 12:59:45,949 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 12:59:45,949 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 12:59:45,949 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-17 12:59:45,988 INFO [regionserver/jenkins-hbase17:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-17 12:59:45,992 INFO [regionserver/jenkins-hbase17:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-17 12:59:45,992 INFO [regionserver/jenkins-hbase17:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-17 12:59:46,147 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740, 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:59:46,348 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740, 7a30f634bdc6e3e5b973c8313abab69a 2023-04-17 12:59:46,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/.tmp/info/79ea19a72e6546289983f592752a6c07 2023-04-17 12:59:46,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/info/0acabc2a158c493388fc5809d2204c2f 2023-04-17 12:59:46,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/.tmp/info/79ea19a72e6546289983f592752a6c07 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/info/79ea19a72e6546289983f592752a6c07 2023-04-17 12:59:46,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0acabc2a158c493388fc5809d2204c2f 2023-04-17 12:59:46,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/info/79ea19a72e6546289983f592752a6c07, entries=2, sequenceid=6, filesize=4.8 K 2023-04-17 12:59:46,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 7a30f634bdc6e3e5b973c8313abab69a in 439ms, sequenceid=6, compaction requested=false 2023-04-17 12:59:46,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/namespace/7a30f634bdc6e3e5b973c8313abab69a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-17 12:59:46,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:59:46,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 7a30f634bdc6e3e5b973c8313abab69a: 2023-04-17 12:59:46,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681736218539.7a30f634bdc6e3e5b973c8313abab69a. 2023-04-17 12:59:46,548 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 12:59:46,748 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 12:59:46,793 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/rep_barrier/50ea424ce530413ebd6d1427e96649b7 2023-04-17 12:59:46,797 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50ea424ce530413ebd6d1427e96649b7 2023-04-17 12:59:46,948 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-17 12:59:46,949 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-17 12:59:46,949 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 12:59:47,149 DEBUG [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 12:59:47,212 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/table/a20f5874934946558b57b97575a9c868 2023-04-17 12:59:47,215 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a20f5874934946558b57b97575a9c868 2023-04-17 12:59:47,216 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/info/0acabc2a158c493388fc5809d2204c2f as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/info/0acabc2a158c493388fc5809d2204c2f 2023-04-17 12:59:47,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0acabc2a158c493388fc5809d2204c2f 2023-04-17 12:59:47,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/info/0acabc2a158c493388fc5809d2204c2f, entries=22, sequenceid=93, filesize=7.1 K 2023-04-17 12:59:47,221 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/rep_barrier/50ea424ce530413ebd6d1427e96649b7 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/rep_barrier/50ea424ce530413ebd6d1427e96649b7 2023-04-17 12:59:47,225 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 50ea424ce530413ebd6d1427e96649b7 2023-04-17 12:59:47,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/rep_barrier/50ea424ce530413ebd6d1427e96649b7, entries=6, sequenceid=93, filesize=5.5 K 2023-04-17 12:59:47,226 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/.tmp/table/a20f5874934946558b57b97575a9c868 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/table/a20f5874934946558b57b97575a9c868 2023-04-17 12:59:47,230 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a20f5874934946558b57b97575a9c868 2023-04-17 12:59:47,230 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/table/a20f5874934946558b57b97575a9c868, entries=9, sequenceid=93, filesize=5.5 K 2023-04-17 12:59:47,231 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1282ms, sequenceid=93, compaction requested=false 2023-04-17 12:59:47,235 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-17 12:59:47,236 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 12:59:47,244 INFO [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 12:59:47,244 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 12:59:47,244 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase17:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-17 12:59:47,349 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1171): stopping server jenkins-hbase17.apache.org,40215,1681736213280; all regions closed. 2023-04-17 12:59:47,359 DEBUG [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/oldWALs 2023-04-17 12:59:47,359 INFO [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C40215%2C1681736213280.meta:.meta(num 1681736217818) 2023-04-17 12:59:47,366 DEBUG [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/oldWALs 2023-04-17 12:59:47,366 INFO [RS:0;jenkins-hbase17:40215] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase17.apache.org%2C40215%2C1681736213280:(num 1681736217117) 2023-04-17 12:59:47,367 DEBUG [RS:0;jenkins-hbase17:40215] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:47,367 INFO [RS:0;jenkins-hbase17:40215] regionserver.LeaseManager(133): Closed leases 2023-04-17 12:59:47,367 INFO [RS:0;jenkins-hbase17:40215] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase17:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-17 12:59:47,368 INFO [regionserver/jenkins-hbase17:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 12:59:47,369 INFO [RS:0;jenkins-hbase17:40215] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:40215 2023-04-17 12:59:47,376 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 12:59:47,376 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase17.apache.org,40215,1681736213280 2023-04-17 12:59:47,376 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 12:59:47,376 ERROR [Listener at localhost.localdomain/43083-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7da5816a rejected from java.util.concurrent.ThreadPoolExecutor@799a1a06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1631] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 12:59:47,378 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase17.apache.org,40215,1681736213280] 2023-04-17 12:59:47,379 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase17.apache.org,40215,1681736213280; numProcessing=1 2023-04-17 12:59:47,380 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase17.apache.org,40215,1681736213280 already deleted, retry=false 2023-04-17 12:59:47,380 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase17.apache.org,40215,1681736213280 expired; onlineServers=0 2023-04-17 12:59:47,380 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase17.apache.org,44317,1681736211069' ***** 2023-04-17 12:59:47,380 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-17 12:59:47,380 DEBUG [M:0;jenkins-hbase17:44317] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fcbdc6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase17.apache.org/136.243.18.41:0 2023-04-17 12:59:47,380 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegionServer(1145): stopping server jenkins-hbase17.apache.org,44317,1681736211069 2023-04-17 12:59:47,380 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegionServer(1171): stopping server jenkins-hbase17.apache.org,44317,1681736211069; all regions closed. 2023-04-17 12:59:47,380 DEBUG [M:0;jenkins-hbase17:44317] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 12:59:47,380 DEBUG [M:0;jenkins-hbase17:44317] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-17 12:59:47,381 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-17 12:59:47,381 DEBUG [M:0;jenkins-hbase17:44317] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-17 12:59:47,381 INFO [M:0;jenkins-hbase17:44317] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-17 12:59:47,381 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1681736216463] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.small.0-1681736216463,5,FailOnTimeoutGroup] 2023-04-17 12:59:47,381 DEBUG [master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1681736216462] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase17:0:becomeActiveMaster-HFileCleaner.large.0-1681736216462,5,FailOnTimeoutGroup] 2023-04-17 12:59:47,381 INFO [M:0;jenkins-hbase17:44317] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-17 12:59:47,381 INFO [M:0;jenkins-hbase17:44317] hbase.ChoreService(369): Chore service for: master/jenkins-hbase17:0 had [] on shutdown 2023-04-17 12:59:47,382 DEBUG [M:0;jenkins-hbase17:44317] master.HMaster(1512): Stopping service threads 2023-04-17 12:59:47,382 INFO [M:0;jenkins-hbase17:44317] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-17 12:59:47,383 ERROR [M:0;jenkins-hbase17:44317] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (257141285) connection to localhost.localdomain/127.0.0.1:38149 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-17 12:59:47,383 INFO [M:0;jenkins-hbase17:44317] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-17 12:59:47,384 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-17 12:59:47,384 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-17 12:59:47,384 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 12:59:47,384 DEBUG [M:0;jenkins-hbase17:44317] zookeeper.ZKUtil(398): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-17 12:59:47,384 WARN [M:0;jenkins-hbase17:44317] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-17 12:59:47,384 INFO [M:0;jenkins-hbase17:44317] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-17 12:59:47,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 12:59:47,385 INFO [M:0;jenkins-hbase17:44317] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-17 12:59:47,385 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 12:59:47,385 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:59:47,385 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:59:47,385 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 12:59:47,385 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:59:47,385 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.33 KB heapSize=393.97 KB 2023-04-17 12:59:47,412 INFO [M:0;jenkins-hbase17:44317] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.33 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/093420eabdaa47a59aa402bdba7cb734 2023-04-17 12:59:47,416 INFO [M:0;jenkins-hbase17:44317] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 093420eabdaa47a59aa402bdba7cb734 2023-04-17 12:59:47,416 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/093420eabdaa47a59aa402bdba7cb734 as hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/093420eabdaa47a59aa402bdba7cb734 2023-04-17 12:59:47,420 INFO [M:0;jenkins-hbase17:44317] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 093420eabdaa47a59aa402bdba7cb734 2023-04-17 12:59:47,420 INFO [M:0;jenkins-hbase17:44317] regionserver.HStore(1080): Added hdfs://localhost.localdomain:38149/user/jenkins/test-data/11058d32-3d93-2012-0ac6-b7610984ba16/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/093420eabdaa47a59aa402bdba7cb734, entries=65, sequenceid=578, filesize=24.7 K 2023-04-17 12:59:47,421 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegion(2948): Finished flush of dataSize ~345.33 KB/353618, heapSize ~393.95 KB/403408, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 36ms, sequenceid=578, compaction requested=false 2023-04-17 12:59:47,422 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 12:59:47,422 DEBUG [M:0;jenkins-hbase17:44317] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 12:59:47,430 INFO [M:0;jenkins-hbase17:44317] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-17 12:59:47,430 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 12:59:47,431 INFO [M:0;jenkins-hbase17:44317] ipc.NettyRpcServer(158): Stopping server on /136.243.18.41:44317 2023-04-17 12:59:47,479 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 12:59:47,479 ERROR [Listener at localhost.localdomain/43083-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c11b475 rejected from java.util.concurrent.ThreadPoolExecutor@799a1a06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1631] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 12:59:47,479 INFO [RS:0;jenkins-hbase17:40215] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase17.apache.org,40215,1681736213280; zookeeper connection closed. 2023-04-17 12:59:47,480 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): regionserver:40215-0x1019766c69b0001, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 12:59:47,480 ERROR [Listener at localhost.localdomain/43083-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@240db1cd rejected from java.util.concurrent.ThreadPoolExecutor@799a1a06[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1631] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 12:59:47,481 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@679e95a3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@679e95a3 2023-04-17 12:59:47,481 INFO [Listener at localhost.localdomain/43083] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-17 12:59:47,586 DEBUG [M:0;jenkins-hbase17:44317] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase17.apache.org,44317,1681736211069 already deleted, retry=false 2023-04-17 12:59:47,823 INFO [M:0;jenkins-hbase17:44317] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase17.apache.org,44317,1681736211069; zookeeper connection closed. 2023-04-17 12:59:47,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 12:59:47,824 ERROR [Listener at localhost.localdomain/43083-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7065d18f rejected from java.util.concurrent.ThreadPoolExecutor@149180c4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2711] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 12:59:47,824 DEBUG [Listener at localhost.localdomain/43083-EventThread] zookeeper.ZKWatcher(600): master:44317-0x1019766c69b0000, quorum=127.0.0.1:62647, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 12:59:47,824 ERROR [Listener at localhost.localdomain/43083-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@492d6008 rejected from java.util.concurrent.ThreadPoolExecutor@149180c4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2711] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 12:59:47,826 WARN [Listener at localhost.localdomain/43083] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-17 12:59:47,834 INFO [Listener at localhost.localdomain/43083] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-17 12:59:47,947 WARN [BP-1294816261-136.243.18.41-1681736208088 heartbeating to localhost.localdomain/127.0.0.1:38149] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-17 12:59:47,948 WARN [BP-1294816261-136.243.18.41-1681736208088 heartbeating to localhost.localdomain/127.0.0.1:38149] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1294816261-136.243.18.41-1681736208088 (Datanode Uuid fabd7e5e-3182-46cc-af5f-125210f1f6a9) service to localhost.localdomain/127.0.0.1:38149 2023-04-17 12:59:47,954 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f/dfs/data/data1/current/BP-1294816261-136.243.18.41-1681736208088] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 12:59:47,954 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c1ddcd5a-e688-0729-8dcd-bd38d139f507/cluster_308a4308-956f-f2fd-ea10-9f4402bb328f/dfs/data/data2/current/BP-1294816261-136.243.18.41-1681736208088] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 12:59:48,029 INFO [Listener at localhost.localdomain/43083] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-17 12:59:48,122 INFO [Listener at localhost.localdomain/43083] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-17 12:59:48,338 INFO [Listener at localhost.localdomain/43083] hbase.HBaseTestingUtility(1293): Minicluster is down